[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:14 dist-test-slave-j4c5 krb5kdc[5423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:14 dist-test-slave-j4c5 krb5kdc[5423](info): set up 2 sockets
Sep 23 23:05:14 dist-test-slave-j4c5 krb5kdc[5423](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 23:05:15.733237 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:15 dist-test-slave-j4c5 krb5kdc[5423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:15.872615 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35741
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:35741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:15.960899 5436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:15.961411 5436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:15.961489 5436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:15.962966 5436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:15.963056 5436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:15.963081 5436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:15.963104 5436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:15.963133 5436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:15.965242 5436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:35741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:15.965613 5436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:15.965919 5436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:15.973182 5441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:15.973182 5442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:15.973210 5444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:15.973520 5436 server_base.cc:1034] running on GCE node
I20260923 23:05:15.973840 5436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:15.974175 5436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:15.975376 5436 hybrid_clock.cc:648] HybridClock initialized: now 1790204715975305 us; error 83 us; skew 500 ppm
Sep 23 23:05:15 dist-test-slave-j4c5 krb5kdc[5423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204715, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:15.984428 5436 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:15.984957 5436 webserver.cc:460] Webserver started at http://127.5.74.62:44791/ using document root <none> and password file <none>
I20260923 23:05:15.985231 5436 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:15.985307 5436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:15.985477 5436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:15.987058 5436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "4062fab817844e96ba0d37b03dcdcd82"
format_stamp: "Formatted at 2026-09-23 23:05:15 on dist-test-slave-j4c5"
server_key: "9299e9ebce87ff9c93e3f6ebe7448fbd"
server_key_iv: "5c82e4cba862c1af5d4f838a9dcfc4f6"
server_key_version: "encryptionkey@0"
I20260923 23:05:15.987542 5436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "4062fab817844e96ba0d37b03dcdcd82"
format_stamp: "Formatted at 2026-09-23 23:05:15 on dist-test-slave-j4c5"
server_key: "9299e9ebce87ff9c93e3f6ebe7448fbd"
server_key_iv: "5c82e4cba862c1af5d4f838a9dcfc4f6"
server_key_version: "encryptionkey@0"
I20260923 23:05:15.989326 5436 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260923 23:05:15.990329 5451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:15.990634 5436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:15.990772 5436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "4062fab817844e96ba0d37b03dcdcd82"
format_stamp: "Formatted at 2026-09-23 23:05:15 on dist-test-slave-j4c5"
server_key: "9299e9ebce87ff9c93e3f6ebe7448fbd"
server_key_iv: "5c82e4cba862c1af5d4f838a9dcfc4f6"
server_key_version: "encryptionkey@0"
I20260923 23:05:15.991003 5436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:16.016901 5436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:16.020017 5436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:16.020344 5436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:16.025610 5436 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:35741
I20260923 23:05:16.025655 5503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:35741 every 8 connection(s)
I20260923 23:05:16.026077 5436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:16.027108 5504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:16.028854 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 5436
I20260923 23:05:16.029076 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:16.029853 5504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82: Bootstrap starting.
I20260923 23:05:16.029896 5416 external_mini_cluster.cc:1428] Setting key b8b3c3c1e4add5b6b9c9dcc1cd6ea597
I20260923 23:05:16.030602 5504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:16.030957 5504 log.cc:824] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:16.032058 5504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82: No bootstrap required, opened a new log
I20260923 23:05:16.034780 5504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4062fab817844e96ba0d37b03dcdcd82" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35741 } }
I20260923 23:05:16.035009 5504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:16.035063 5504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4062fab817844e96ba0d37b03dcdcd82, State: Initialized, Role: FOLLOWER
I20260923 23:05:16.035231 5504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4062fab817844e96ba0d37b03dcdcd82" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35741 } }
I20260923 23:05:16.035441 5504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:16.035578 5504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:16.035641 5504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:16.037160 5504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4062fab817844e96ba0d37b03dcdcd82" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35741 } }
I20260923 23:05:16.037338 5504 leader_election.cc:302] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4062fab817844e96ba0d37b03dcdcd82; no voters:
I20260923 23:05:16.037648 5504 leader_election.cc:288] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:16.037810 5509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:16.037949 5504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:16.037958 5509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [term 1 LEADER]: Becoming Leader. State: Replica: 4062fab817844e96ba0d37b03dcdcd82, State: Running, Role: LEADER
I20260923 23:05:16.038081 5509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4062fab817844e96ba0d37b03dcdcd82" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35741 } }
I20260923 23:05:16.039146 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4062fab817844e96ba0d37b03dcdcd82. Latest consensus state: current_term: 1 leader_uuid: "4062fab817844e96ba0d37b03dcdcd82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4062fab817844e96ba0d37b03dcdcd82" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35741 } } }
I20260923 23:05:16.039242 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:16.039376 5510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4062fab817844e96ba0d37b03dcdcd82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4062fab817844e96ba0d37b03dcdcd82" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35741 } } }
I20260923 23:05:16.039436 5510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:16.039543 5518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:16.040082 5518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:16.042716 5518 catalog_manager.cc:1347] Generated new cluster ID: 91a4b88a296541148f969a8bad732716
I20260923 23:05:16.042794 5518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:16.043313 5526 mvcc.cc:205] Tried to move back new op lower bound from 7332678516904235008 to 7332678516892119040. Current Snapshot: MvccSnapshot[applied={T|T < 7332678516904235008 or (T in {7332678516904235008})}]
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:16.050920 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:16.030863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.5.74.62:35741)
0923 23:05:16.031194 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:16.031206 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:16.031481 (+ 275us) server_negotiation.cc:407] Connection header received
0923 23:05:16.034207 (+ 2726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:16.034217 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:16.034291 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:16.034493 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:16.036722 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.039057 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.041172 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.041822 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.041942 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:16.046762 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:16.046766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:16.046766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:16.046778 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:16.048876 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.049372 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.049376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.049376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.049480 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.049776 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.049777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.049778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.050042 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:16.050221 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:16.050509 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 23:05:16.050667 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
I20260923 23:05:16.062845 5518 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:16.063493 5518 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:16.068413 5518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4062fab817844e96ba0d37b03dcdcd82: Generated new TSK 0
I20260923 23:05:16.068656 5518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:16.105713 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35741
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:16.198083 5530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:16.198326 5530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:16.198362 5530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:16.199810 5530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:16.199882 5530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:16.199925 5530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:16.201848 5530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:16.202157 5530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:16.202473 5530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:16.208063 5535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:16.208123 5538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:16.208130 5530 server_base.cc:1034] running on GCE node
W20260923 23:05:16.208374 5536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:16.208642 5530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:16.208878 5530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:16.210029 5530 hybrid_clock.cc:648] HybridClock initialized: now 1790204716210014 us; error 27 us; skew 500 ppm
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:16.212872 5530 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:16.213356 5530 webserver.cc:460] Webserver started at http://127.5.74.1:45439/ using document root <none> and password file <none>
I20260923 23:05:16.213590 5530 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:16.213644 5530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:16.213780 5530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:16.214911 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "dccb50f7021f4497be4c2551ceb05159"
format_stamp: "Formatted at 2026-09-23 23:05:16 on dist-test-slave-j4c5"
server_key: "6628dab06f88c5b9fa6eb0f9a18b1e6b"
server_key_iv: "780d736fca8ae24c0f68dfe1b7283bce"
server_key_version: "encryptionkey@0"
I20260923 23:05:16.215273 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "dccb50f7021f4497be4c2551ceb05159"
format_stamp: "Formatted at 2026-09-23 23:05:16 on dist-test-slave-j4c5"
server_key: "6628dab06f88c5b9fa6eb0f9a18b1e6b"
server_key_iv: "780d736fca8ae24c0f68dfe1b7283bce"
server_key_version: "encryptionkey@0"
I20260923 23:05:16.216905 5530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:16.217849 5545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:16.218116 5530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:16.218199 5530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "dccb50f7021f4497be4c2551ceb05159"
format_stamp: "Formatted at 2026-09-23 23:05:16 on dist-test-slave-j4c5"
server_key: "6628dab06f88c5b9fa6eb0f9a18b1e6b"
server_key_iv: "780d736fca8ae24c0f68dfe1b7283bce"
server_key_version: "encryptionkey@0"
I20260923 23:05:16.218307 5530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:16.228809 5530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:16.231341 5530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:16.231514 5530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:16.231786 5530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:16.232240 5530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:16.232280 5530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:16.232304 5530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:16.232343 5530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:16.240653 5530 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:43281
I20260923 23:05:16.240783 5658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:43281 every 8 connection(s)
I20260923 23:05:16.241245 5530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:16.241483 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 5530
I20260923 23:05:16.241608 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:16.241792 5416 external_mini_cluster.cc:1428] Setting key 4c02f09a45a2ef93d0449ad38ba13441
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:05:16.253875 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:16.242287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:37113 (local address 127.5.74.62:35741)
0923 23:05:16.242338 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:16.242340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:16.242645 (+ 305us) server_negotiation.cc:407] Connection header received
0923 23:05:16.243222 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:16.243224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:16.243238 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:16.243321 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:16.244632 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.245198 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.246232 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.246448 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.246538 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:16.249520 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:16.249528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:16.249528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:16.249535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:16.251794 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.252539 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.252541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.252542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.252612 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.252903 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.252905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.252905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.253128 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:16.253203 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:16.253657 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 23:05:16.253762 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:05:16.254531 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:16.242308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35741 (local address 127.5.74.1:37113)
0923 23:05:16.242503 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:16.242551 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:16.243129 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:16.243384 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:16.243388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:16.244119 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:16.244507 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:16.244514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.245302 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.245303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:16.246132 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:16.246135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.246548 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.246549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:16.246812 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:16.247500 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:16.247510 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:16.249385 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:16.251929 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:16.251940 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:16.251944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:16.252434 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:16.252696 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:16.252697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:16.252697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:16.252825 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:16.253243 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:16.253250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:16.253401 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 23:05:16.253513 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 23:05:16.253977 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260923 23:05:16.255489 5659 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35741
I20260923 23:05:16.255650 5659 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
I20260923 23:05:16.255894 5659 heartbeater.cc:511] Master 127.5.74.62:35741 requested a full tablet report, sending...
I20260923 23:05:16.256958 5468 ts_manager.cc:194] Registered new tserver with Master: dccb50f7021f4497be4c2551ceb05159 (127.5.74.1:43281)
I20260923 23:05:16.257858 5468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:37113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:16.277637 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35741
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:16.369158 5664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:16.369441 5664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:16.369485 5664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:16.370769 5664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:16.370862 5664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:16.370915 5664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:16.372541 5664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:16.372833 5664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:16.373093 5664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:16.378913 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:16.378945 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:16.378981 5664 server_base.cc:1034] running on GCE node
W20260923 23:05:16.378934 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:16.379398 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:16.379666 5664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:16.380826 5664 hybrid_clock.cc:648] HybridClock initialized: now 1790204716380808 us; error 48 us; skew 500 ppm
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:16.384110 5664 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:16.384649 5664 webserver.cc:460] Webserver started at http://127.5.74.2:34089/ using document root <none> and password file <none>
I20260923 23:05:16.384899 5664 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:16.384971 5664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:16.385135 5664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:16.386245 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "044757b8ecd74be2991f395404e2ec58"
format_stamp: "Formatted at 2026-09-23 23:05:16 on dist-test-slave-j4c5"
server_key: "97d65a8982d3f23b6647e8231519519a"
server_key_iv: "7e79c95285a7401429fda0e887564b65"
server_key_version: "encryptionkey@0"
I20260923 23:05:16.386655 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "044757b8ecd74be2991f395404e2ec58"
format_stamp: "Formatted at 2026-09-23 23:05:16 on dist-test-slave-j4c5"
server_key: "97d65a8982d3f23b6647e8231519519a"
server_key_iv: "7e79c95285a7401429fda0e887564b65"
server_key_version: "encryptionkey@0"
I20260923 23:05:16.388652 5664 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 23:05:16.389945 5679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:16.390192 5664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:16.390431 5664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "044757b8ecd74be2991f395404e2ec58"
format_stamp: "Formatted at 2026-09-23 23:05:16 on dist-test-slave-j4c5"
server_key: "97d65a8982d3f23b6647e8231519519a"
server_key_iv: "7e79c95285a7401429fda0e887564b65"
server_key_version: "encryptionkey@0"
I20260923 23:05:16.390566 5664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:16.402420 5664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:16.405059 5664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:16.405241 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:16.405500 5664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:16.405857 5664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:16.405915 5664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:16.405959 5664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:16.405987 5664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:16.414867 5664 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:37947
I20260923 23:05:16.414899 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:37947 every 8 connection(s)
I20260923 23:05:16.415426 5664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:16.423692 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 5664
I20260923 23:05:16.423818 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:16.423996 5416 external_mini_cluster.cc:1428] Setting key bdfc70a3a8f9d8114c6dc2093f337bb0
I20260923 23:05:16.427419 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:16.416284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:40791 (local address 127.5.74.62:35741)
0923 23:05:16.416399 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:16.416402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:16.416558 (+ 156us) server_negotiation.cc:407] Connection header received
0923 23:05:16.417114 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:16.417115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:16.417128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:16.417183 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:16.418181 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.418693 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.419760 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.419976 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.420065 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:16.422714 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:16.422716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:16.422717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:16.422724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:16.424873 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.425817 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.425818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.425819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.425884 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.426261 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.426262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.426263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.426393 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:16.426702 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:16.427164 (+ 462us) server_negotiation.cc:299] Negotiation successful
0923 23:05:16.427275 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 23:05:16.427459 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:16.416229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35741 (local address 127.5.74.2:40791)
0923 23:05:16.416418 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:16.416474 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:16.417001 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:16.417273 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:16.417277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:16.417870 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:16.418070 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:16.418075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.418794 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.418795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:16.419656 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:16.419659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.420078 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.420079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:16.420227 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:16.420816 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:16.420825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:16.422610 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:16.425160 (+ 2550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:16.425172 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:16.425175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:16.425693 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:16.425966 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:16.425967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:16.425967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:16.426129 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:16.426704 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:16.426712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:16.426983 (+ 271us) client_negotiation.cc:769] Sending connection context
0923 23:05:16.427112 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 23:05:16.427278 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260923 23:05:16.428289 5793 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35741
I20260923 23:05:16.428395 5793 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:16.428606 5793 heartbeater.cc:511] Master 127.5.74.62:35741 requested a full tablet report, sending...
I20260923 23:05:16.429098 5468 ts_manager.cc:194] Registered new tserver with Master: 044757b8ecd74be2991f395404e2ec58 (127.5.74.2:37947)
I20260923 23:05:16.429528 5468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:40791
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:16.456015 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35741
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:16.553817 5798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:16.554059 5798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:16.554087 5798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:16.555301 5798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:16.555347 5798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:16.555377 5798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:16.557184 5798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:16.557525 5798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:16.557776 5798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:16.563133 5803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:16.563120 5806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:16.563315 5804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:16.563773 5798 server_base.cc:1034] running on GCE node
I20260923 23:05:16.563967 5798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:16.564232 5798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:16.565450 5798 hybrid_clock.cc:648] HybridClock initialized: now 1790204716565442 us; error 46 us; skew 500 ppm
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:16.568094 5798 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:16.568728 5798 webserver.cc:460] Webserver started at http://127.5.74.3:41933/ using document root <none> and password file <none>
I20260923 23:05:16.568979 5798 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:16.569051 5798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:16.569212 5798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:16.570335 5798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "b8ee5d9dd65f434aa864aada9f5e6e18"
format_stamp: "Formatted at 2026-09-23 23:05:16 on dist-test-slave-j4c5"
server_key: "28e9aea66442f6d68957de91cce34851"
server_key_iv: "3c2d0b3b6a8e5b248b5ee2dedf023146"
server_key_version: "encryptionkey@0"
I20260923 23:05:16.570720 5798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "b8ee5d9dd65f434aa864aada9f5e6e18"
format_stamp: "Formatted at 2026-09-23 23:05:16 on dist-test-slave-j4c5"
server_key: "28e9aea66442f6d68957de91cce34851"
server_key_iv: "3c2d0b3b6a8e5b248b5ee2dedf023146"
server_key_version: "encryptionkey@0"
I20260923 23:05:16.572350 5798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260923 23:05:16.573182 5813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:16.573514 5798 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:16.573654 5798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "b8ee5d9dd65f434aa864aada9f5e6e18"
format_stamp: "Formatted at 2026-09-23 23:05:16 on dist-test-slave-j4c5"
server_key: "28e9aea66442f6d68957de91cce34851"
server_key_iv: "3c2d0b3b6a8e5b248b5ee2dedf023146"
server_key_version: "encryptionkey@0"
I20260923 23:05:16.573735 5798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:16.598493 5798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:16.601778 5798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:16.601992 5798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:16.602356 5798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:16.602840 5798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:16.602902 5798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:16.602938 5798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:16.602955 5798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:16.613159 5798 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:40157
I20260923 23:05:16.613205 5926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:40157 every 8 connection(s)
I20260923 23:05:16.613569 5798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:16.623054 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 5798
I20260923 23:05:16.623168 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:16.623656 5416 external_mini_cluster.cc:1428] Setting key 02c3848c4e68dcfca37df4bbe6c9627b
I20260923 23:05:16.625972 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:16.614523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:57525 (local address 127.5.74.62:35741)
0923 23:05:16.614597 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:16.614599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:16.614854 (+ 255us) server_negotiation.cc:407] Connection header received
0923 23:05:16.615456 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:16.615457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:16.615469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:16.615538 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:16.616607 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.617143 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.618456 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.618656 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.618728 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:16.621574 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:16.621576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:16.621576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:16.621584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:16.624040 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.624752 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.624752 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.624753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.624939 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.625258 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.625259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.625260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.625403 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:16.625460 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:16.625811 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 23:05:16.625886 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 23:05:16.626060 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:16.614493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35741 (local address 127.5.74.3:57525)
0923 23:05:16.614672 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:16.614713 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:16.615280 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:16.615637 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:16.615642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:16.616282 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:16.616494 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:16.616499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.617251 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.617252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:16.618353 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:16.618356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.618740 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.618740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:16.618843 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:16.619460 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:16.619472 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:16.621441 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:16.624222 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:16.624231 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:16.624235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:16.624660 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:16.625021 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:16.625022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:16.625022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:16.625173 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:16.625485 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:16.625491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:16.625620 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 23:05:16.625705 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 23:05:16.625843 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260923 23:05:16.626986 5927 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35741
I20260923 23:05:16.627223 5927 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:16.627490 5927 heartbeater.cc:511] Master 127.5.74.62:35741 requested a full tablet report, sending...
I20260923 23:05:16.628393 5468 ts_manager.cc:194] Registered new tserver with Master: b8ee5d9dd65f434aa864aada9f5e6e18 (127.5.74.3:40157)
I20260923 23:05:16.629195 5468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:57525
I20260923 23:05:16.637681 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:16.664883 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:16.655300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.5.74.1:43281)
0923 23:05:16.655473 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:16.655478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:16.655505 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:05:16.655531 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:16.655533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:16.655547 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:16.655686 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:16.656536 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.657308 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.658162 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.658540 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.658602 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:16.661245 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:16.661255 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:16.661255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:16.661263 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:16.663406 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.663782 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.663786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.663787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.663854 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.664074 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.664075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.664076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.664387 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:16.664535 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:16.664716 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 23:05:16.664793 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
W20260923 23:05:16.665407 5573 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59206
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:16.690639 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:16.682437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.5.74.1:43281)
0923 23:05:16.682533 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:16.682536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:16.682555 (+ 19us) server_negotiation.cc:407] Connection header received
0923 23:05:16.682630 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:16.682631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:16.682643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:16.682695 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:16.683340 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.683919 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.684834 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:16.685064 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:16.685140 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:16.687698 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:16.687699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:16.687700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:16.687707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:16.689415 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.689837 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.689838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.689839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.689904 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:16.690128 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:16.690130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:16.690130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:16.690239 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:16.690364 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:16.690506 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 23:05:16.690571 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 23:05:16.691326 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 5530
I20260923 23:05:16.696290 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 5664
I20260923 23:05:16.701562 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 5798
I20260923 23:05:16.706061 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 5436
2026-09-23T23:05:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2076 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5941](info): set up 2 sockets
Sep 23 23:05:16 dist-test-slave-j4c5 krb5kdc[5941](info): commencing operation
krb5kdc: starting...
W20260923 23:05:17.791822 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:17 dist-test-slave-j4c5 krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:17.924299 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35175
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:18.014976 5954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:18.015250 5954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:18.015295 5954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:18.016568 5954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:18.016647 5954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:18.016672 5954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:18.016693 5954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:18.016724 5954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:18.018285 5954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:35175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:18.018586 5954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:18.018826 5954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:18.024309 5962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:18.024526 5954 server_base.cc:1034] running on GCE node
W20260923 23:05:18.024329 5959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:18.024384 5960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:18.024847 5954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:18.025173 5954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:18.026383 5954 hybrid_clock.cc:648] HybridClock initialized: now 1790204718026351 us; error 49 us; skew 500 ppm
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:18.029347 5954 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:18.029771 5954 webserver.cc:460] Webserver started at http://127.5.74.62:35791/ using document root <none> and password file <none>
I20260923 23:05:18.030010 5954 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:18.030059 5954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:18.030195 5954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:18.031301 5954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "87c6a3c4db464671a4d401b6000dcc6f"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "720c574a58c57bf99ddf9c0c5fa65c46"
server_key_iv: "9aa25979bd53b1ab3eb5275754ff9598"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.031679 5954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "87c6a3c4db464671a4d401b6000dcc6f"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "720c574a58c57bf99ddf9c0c5fa65c46"
server_key_iv: "9aa25979bd53b1ab3eb5275754ff9598"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.033234 5954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:05:18.034161 5969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.034417 5954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:18.034539 5954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "87c6a3c4db464671a4d401b6000dcc6f"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "720c574a58c57bf99ddf9c0c5fa65c46"
server_key_iv: "9aa25979bd53b1ab3eb5275754ff9598"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.034614 5954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:18.064710 5954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:18.067389 5954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:18.067529 5954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:18.074509 5954 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:35175
I20260923 23:05:18.074564 6021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:35175 every 8 connection(s)
I20260923 23:05:18.074965 5954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:18.075954 6022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:18.078351 6022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f: Bootstrap starting.
I20260923 23:05:18.078951 6022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:18.079198 6022 log.cc:824] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:18.079941 6022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f: No bootstrap required, opened a new log
I20260923 23:05:18.080236 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 5954
I20260923 23:05:18.080327 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:18.080477 5416 external_mini_cluster.cc:1428] Setting key 58267d6072ef51d3b7f5b626758c766c
I20260923 23:05:18.082093 6022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c6a3c4db464671a4d401b6000dcc6f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35175 } }
I20260923 23:05:18.082293 6022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:18.082341 6022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c6a3c4db464671a4d401b6000dcc6f, State: Initialized, Role: FOLLOWER
I20260923 23:05:18.082479 6022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c6a3c4db464671a4d401b6000dcc6f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35175 } }
I20260923 23:05:18.082574 6022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:18.082610 6022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:18.082653 6022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:18.083787 6022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c6a3c4db464671a4d401b6000dcc6f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35175 } }
I20260923 23:05:18.083920 6022 leader_election.cc:302] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c6a3c4db464671a4d401b6000dcc6f; no voters:
I20260923 23:05:18.084129 6022 leader_election.cc:288] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:18.084184 6027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:18.084334 6027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [term 1 LEADER]: Becoming Leader. State: Replica: 87c6a3c4db464671a4d401b6000dcc6f, State: Running, Role: LEADER
I20260923 23:05:18.084438 6022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:18.084474 6027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c6a3c4db464671a4d401b6000dcc6f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35175 } }
I20260923 23:05:18.084899 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c6a3c4db464671a4d401b6000dcc6f. Latest consensus state: current_term: 1 leader_uuid: "87c6a3c4db464671a4d401b6000dcc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c6a3c4db464671a4d401b6000dcc6f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35175 } } }
I20260923 23:05:18.085031 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:18.085359 6029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c6a3c4db464671a4d401b6000dcc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c6a3c4db464671a4d401b6000dcc6f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35175 } } }
I20260923 23:05:18.085439 6029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f [sys.catalog]: This master's current role is: LEADER
W20260923 23:05:18.086501 6043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:05:18.086926 6043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:05:18.086988 6034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:18.087145 6034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:18.089082 6034 catalog_manager.cc:1347] Generated new cluster ID: df06bb1f3a6d4da2817141e3da9835ba
I20260923 23:05:18.089154 6034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:18.095079 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.081023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.5.74.62:35175)
0923 23:05:18.081226 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.081235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.081262 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:05:18.082184 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.082193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.082262 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.082448 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:18.083273 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.084464 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.085567 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.086195 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.086332 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.089484 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:18.089488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:18.089489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:18.089502 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:18.092136 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.092720 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.092723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.092724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.092860 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.093264 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.093265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.093265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.093754 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:18.094545 (+ 791us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.094667 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.094844 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260923 23:05:18.101203 6034 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:18.102100 6034 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:18.113715 6034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c6a3c4db464671a4d401b6000dcc6f: Generated new TSK 0
I20260923 23:05:18.113961 6034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:18.143937 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35175
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:18.233086 6048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:18.233366 6048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:18.233412 6048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:18.234783 6048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:18.234870 6048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:18.234925 6048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:18.236825 6048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:18.237149 6048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:18.237422 6048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:18.243415 6054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:18.243435 6053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:18.243479 6056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:18.243655 6048 server_base.cc:1034] running on GCE node
I20260923 23:05:18.243896 6048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:18.244222 6048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:18.245400 6048 hybrid_clock.cc:648] HybridClock initialized: now 1790204718245349 us; error 63 us; skew 500 ppm
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:18.248234 6048 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:18.248755 6048 webserver.cc:460] Webserver started at http://127.5.74.1:45481/ using document root <none> and password file <none>
I20260923 23:05:18.249002 6048 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:18.249090 6048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:18.249262 6048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:18.250440 6048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "e6954cb9d32743b185b740f00781c57c"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "0847651880f6f92c51640da040ffcca4"
server_key_iv: "764ee3da66a26263e5335d61939052a0"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.250834 6048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "e6954cb9d32743b185b740f00781c57c"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "0847651880f6f92c51640da040ffcca4"
server_key_iv: "764ee3da66a26263e5335d61939052a0"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.252556 6048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260923 23:05:18.253513 6063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.253760 6048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:18.253887 6048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "e6954cb9d32743b185b740f00781c57c"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "0847651880f6f92c51640da040ffcca4"
server_key_iv: "764ee3da66a26263e5335d61939052a0"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.253981 6048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:18.273739 6048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:18.276608 6048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:18.276821 6048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:18.277101 6048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:18.277531 6048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:18.277621 6048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.277678 6048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:18.277726 6048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.285843 6048 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:34755
I20260923 23:05:18.285880 6176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:34755 every 8 connection(s)
I20260923 23:05:18.286314 6048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:18.289881 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 6048
I20260923 23:05:18.289983 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:18.290246 5416 external_mini_cluster.cc:1428] Setting key 226d4f32aadcd3067b4e278a6ad5e68e
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:05:18.299003 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.287630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:47011 (local address 127.5.74.62:35175)
0923 23:05:18.287678 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.287680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.287962 (+ 282us) server_negotiation.cc:407] Connection header received
0923 23:05:18.289323 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.289324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.289339 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.289396 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:18.290449 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.291105 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.292576 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.292920 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.293091 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.295942 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:18.295944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:18.295944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:18.295960 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:18.297490 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.297968 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.297970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.297970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.298036 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.298315 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.298316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.298316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.298438 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:18.298510 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.298809 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.298902 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:18.299217 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.287527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35175 (local address 127.5.74.1:47011)
0923 23:05:18.287802 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:18.287851 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:18.289096 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:18.289484 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:18.289488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:18.289894 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:18.290326 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.290335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.291209 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.291210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.292476 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.292478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.293107 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.293108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.293335 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.293966 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:18.293978 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:18.295827 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:18.297608 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:18.297617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:18.297620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:18.297882 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:18.298106 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:18.298106 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:18.298107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:18.298232 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:18.298512 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:18.298519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:18.298673 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 23:05:18.298795 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 23:05:18.298965 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260923 23:05:18.299744 6177 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35175
I20260923 23:05:18.299835 6177 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:18.300062 6177 heartbeater.cc:511] Master 127.5.74.62:35175 requested a full tablet report, sending...
I20260923 23:05:18.300680 5986 ts_manager.cc:194] Registered new tserver with Master: e6954cb9d32743b185b740f00781c57c (127.5.74.1:34755)
I20260923 23:05:18.301285 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:47011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:18.320013 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35175
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:18.414047 6182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:18.414309 6182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:18.414340 6182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:18.415763 6182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:18.415817 6182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:18.415851 6182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:18.418033 6182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:18.418514 6182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:18.418845 6182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:18.424736 6188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:18.424811 6187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:18.424950 6190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:18.425400 6182 server_base.cc:1034] running on GCE node
I20260923 23:05:18.425796 6182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:18.426098 6182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:18.427291 6182 hybrid_clock.cc:648] HybridClock initialized: now 1790204718427256 us; error 51 us; skew 500 ppm
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:18.430780 6182 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:18.431385 6182 webserver.cc:460] Webserver started at http://127.5.74.2:43069/ using document root <none> and password file <none>
I20260923 23:05:18.431677 6182 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:18.431757 6182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:18.431936 6182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:18.433286 6182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "ff4025f5800d42469b081c6d57d6f50c"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "1cefc681278d37ccc1f0debd6cca7cfa"
server_key_iv: "b353910b4b77a1b2929d1f43b1fad751"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.433704 6182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "ff4025f5800d42469b081c6d57d6f50c"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "1cefc681278d37ccc1f0debd6cca7cfa"
server_key_iv: "b353910b4b77a1b2929d1f43b1fad751"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.435227 6182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 23:05:18.436196 6197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.436441 6182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:18.436560 6182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "ff4025f5800d42469b081c6d57d6f50c"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "1cefc681278d37ccc1f0debd6cca7cfa"
server_key_iv: "b353910b4b77a1b2929d1f43b1fad751"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.436666 6182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:18.451005 6182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:18.453614 6182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:18.453914 6182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:18.454224 6182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:18.454612 6182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:18.454671 6182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.454710 6182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:18.454766 6182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.461755 6182 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:38425
I20260923 23:05:18.461797 6310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:38425 every 8 connection(s)
I20260923 23:05:18.462231 6182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:05:18.466135 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 6182
I20260923 23:05:18.466310 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:18.466506 5416 external_mini_cluster.cc:1428] Setting key 36c5ecab0da71de6ebdaf49746e056d0
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:18.474692 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.463109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:38515 (local address 127.5.74.62:35175)
0923 23:05:18.463290 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.463292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.463682 (+ 390us) server_negotiation.cc:407] Connection header received
0923 23:05:18.464827 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.464829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.464842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.464911 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:18.465651 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.466150 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.467438 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.467635 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.467745 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.471069 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:18.471071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:18.471071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:18.471079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:18.472784 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.473450 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.473452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.473453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.473522 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.473919 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.473921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.473921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.474055 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:18.474136 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.474530 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.474616 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260923 23:05:18.474970 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.463074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35175 (local address 127.5.74.2:38515)
0923 23:05:18.463540 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:18.463585 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:18.464715 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:18.464993 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:18.465003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:18.465334 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:18.465547 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.465552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.466244 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.466245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.467332 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.467334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.467749 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.467750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.468020 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.468701 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:18.468711 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:18.470944 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:18.472889 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:18.472901 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:18.472904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:18.473321 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:18.473601 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:18.473602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:18.473602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:18.473774 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:18.474146 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:18.474153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:18.474354 (+ 201us) client_negotiation.cc:769] Sending connection context
0923 23:05:18.474485 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 23:05:18.474684 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":160,"threads_started":1}
I20260923 23:05:18.475538 6311 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35175
I20260923 23:05:18.475659 6311 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:18.475880 6311 heartbeater.cc:511] Master 127.5.74.62:35175 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
I20260923 23:05:18.476465 5986 ts_manager.cc:194] Registered new tserver with Master: ff4025f5800d42469b081c6d57d6f50c (127.5.74.2:38425)
I20260923 23:05:18.476898 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:38515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:18.501188 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35175
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:18.589303 6316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:18.589553 6316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:18.589583 6316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:18.590914 6316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:18.590972 6316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:18.591003 6316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:18.592752 6316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:18.593029 6316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:18.593290 6316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:18.598872 6321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:18.598874 6324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:18.598887 6322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:18.599025 6316 server_base.cc:1034] running on GCE node
I20260923 23:05:18.599514 6316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:18.599759 6316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:18.600925 6316 hybrid_clock.cc:648] HybridClock initialized: now 1790204718600894 us; error 44 us; skew 500 ppm
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:18.603884 6316 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:18.604445 6316 webserver.cc:460] Webserver started at http://127.5.74.3:38119/ using document root <none> and password file <none>
I20260923 23:05:18.604708 6316 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:18.604759 6316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:18.604910 6316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:18.606087 6316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "6f34915ac8d0456f809b797116b656b9"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "20336eda7e6e570eeb4325457d1be863"
server_key_iv: "36550e81beadafba60f4f8549b8f6acf"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.606482 6316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "6f34915ac8d0456f809b797116b656b9"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "20336eda7e6e570eeb4325457d1be863"
server_key_iv: "36550e81beadafba60f4f8549b8f6acf"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.608114 6316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:18.609261 6331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.609632 6316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:18.609741 6316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "6f34915ac8d0456f809b797116b656b9"
format_stamp: "Formatted at 2026-09-23 23:05:18 on dist-test-slave-j4c5"
server_key: "20336eda7e6e570eeb4325457d1be863"
server_key_iv: "36550e81beadafba60f4f8549b8f6acf"
server_key_version: "encryptionkey@0"
I20260923 23:05:18.609820 6316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:18.629366 6316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:18.631982 6316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:18.632119 6316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:18.632434 6316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:18.632754 6316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:18.632802 6316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.632826 6316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:18.632839 6316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:18.640086 6316 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:33261
I20260923 23:05:18.640188 6444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:33261 every 8 connection(s)
I20260923 23:05:18.640583 6316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:05:18.647517 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 6316
I20260923 23:05:18.647784 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:18.648180 5416 external_mini_cluster.cc:1428] Setting key 0a1944f054447d24c1690f6f5731c249
I20260923 23:05:18.651764 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.641487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:43187 (local address 127.5.74.62:35175)
0923 23:05:18.641570 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.641572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.641834 (+ 262us) server_negotiation.cc:407] Connection header received
0923 23:05:18.642873 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.642874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.642886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.642972 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:18.643684 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.644330 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.645513 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.645733 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.645812 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.648711 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:18.648713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:18.648713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:18.648722 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:18.650234 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.650751 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.650752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.650753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.650819 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.651079 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.651080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.651081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.651209 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:18.651298 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.651581 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.651664 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 23:05:18.652009 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.641458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35175 (local address 127.5.74.3:43187)
0923 23:05:18.641655 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:18.641704 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:18.642733 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:18.643040 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:18.643044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:18.643359 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:18.643576 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.643581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.644450 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.644452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.645402 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.645405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.645845 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.645847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.645963 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.646652 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:18.646661 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:18.648600 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:18.650404 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:18.650413 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:18.650416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:18.650667 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:18.650889 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:18.650890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:18.650890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:18.651008 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:18.651281 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:18.651288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:18.651443 (+ 155us) client_negotiation.cc:769] Sending connection context
0923 23:05:18.651558 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 23:05:18.651731 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
I20260923 23:05:18.652865 6445 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35175
I20260923 23:05:18.652963 6445 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:18.653126 6445 heartbeater.cc:511] Master 127.5.74.62:35175 requested a full tablet report, sending...
I20260923 23:05:18.653532 5986 ts_manager.cc:194] Registered new tserver with Master: 6f34915ac8d0456f809b797116b656b9 (127.5.74.3:33261)
I20260923 23:05:18.653920 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:43187
I20260923 23:05:18.660732 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:05:18.668897 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.662374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.5.74.62:35175)
0923 23:05:18.662452 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.662454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.662555 (+ 101us) server_negotiation.cc:407] Connection header received
0923 23:05:18.662612 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.662613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.662623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.662670 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:18.663259 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.663794 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.664622 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.664804 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.664862 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.666206 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:18.666207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:18.666208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:18.666214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:18.667543 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.667953 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.667954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.667954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.668004 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.668288 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.668290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.668290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.668463 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:18.668571 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.668708 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.668791 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 23:05:18.671350 5986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:18.673007 5986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:18.680317 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.676254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.5.74.3:33261)
0923 23:05:18.676404 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.676410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.676420 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:18.676630 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.676633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.676652 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.676911 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:18.677011 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.678080 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.679333 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.679826 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.679935 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.680001 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.680149 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.680218 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260923 23:05:18.680306 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.675973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:33261 (local address 127.0.0.1:56202)
0923 23:05:18.676103 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:18.676146 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:18.676235 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:18.676760 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:18.676763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:18.676765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:18.676840 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.676842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.678195 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.678196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.679251 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.679253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.679940 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.679941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.680029 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.680044 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 23:05:18.680118 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 23:05:18.680224 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":47,"thread_start_us":112,"threads_started":1}
I20260923 23:05:18.680478 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.676277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.5.74.2:38425)
0923 23:05:18.676557 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.676561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.676573 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:05:18.676646 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.676649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.676664 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.676743 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:18.676983 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.678107 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.679279 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.679955 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.680011 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.680055 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.680352 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.680397 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260923 23:05:18.680644 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.675973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:38425 (local address 127.0.0.1:58416)
0923 23:05:18.676465 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:18.676476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:18.676560 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:18.676830 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:18.676830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:18.676833 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:18.676892 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.676894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.678211 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.678212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.679186 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.679188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.680117 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.680118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.680216 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.680221 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:18.680272 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:05:18.680309 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":26,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260923 23:05:18.681056 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.675974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:34755 (local address 127.0.0.1:41314)
0923 23:05:18.676423 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:18.676454 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:18.676511 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:18.676917 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:18.676918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:18.676920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:18.676954 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.676956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.677800 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.677801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.679039 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.679041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.680045 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.680046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.680107 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.680110 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:18.680179 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 23:05:18.680995 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":80,"threads_started":1}
I20260923 23:05:18.680352 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.676283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.5.74.1:34755)
0923 23:05:18.676694 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.676698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.676707 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:18.676742 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.676743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.676758 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.676826 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:18.677016 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.677692 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.679133 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.679887 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.679962 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.680022 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.680232 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.680273 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":86,"threads_started":1}
I20260923 23:05:18.681141 6379 tablet_service.cc:1467] Processing CreateTablet for tablet 0e51fc5429a047f091f9648a9edaecd0 (DEFAULT_TABLE table=test-table [id=d63b3fcf2d28420e9f7a2a1db74a80ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:18.681224 6245 tablet_service.cc:1467] Processing CreateTablet for tablet 0e51fc5429a047f091f9648a9edaecd0 (DEFAULT_TABLE table=test-table [id=d63b3fcf2d28420e9f7a2a1db74a80ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:18.681500 6245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e51fc5429a047f091f9648a9edaecd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:18.681500 6379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e51fc5429a047f091f9648a9edaecd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:18.681766 6110 tablet_service.cc:1467] Processing CreateTablet for tablet 0e51fc5429a047f091f9648a9edaecd0 (DEFAULT_TABLE table=test-table [id=d63b3fcf2d28420e9f7a2a1db74a80ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:18.682085 6110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e51fc5429a047f091f9648a9edaecd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:18.684855 6462 tablet_bootstrap.cc:492] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c: Bootstrap starting.
I20260923 23:05:18.685140 6463 tablet_bootstrap.cc:492] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c: Bootstrap starting.
I20260923 23:05:18.685155 6464 tablet_bootstrap.cc:492] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9: Bootstrap starting.
I20260923 23:05:18.685652 6462 tablet_bootstrap.cc:654] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:18.685796 6463 tablet_bootstrap.cc:654] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:18.685796 6464 tablet_bootstrap.cc:654] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:18.686208 6462 log.cc:824] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:18.686398 6464 log.cc:824] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:18.686398 6463 log.cc:824] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:18.687232 6462 tablet_bootstrap.cc:492] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c: No bootstrap required, opened a new log
I20260923 23:05:18.687253 6463 tablet_bootstrap.cc:492] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c: No bootstrap required, opened a new log
I20260923 23:05:18.687314 6463 ts_tablet_manager.cc:1397] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:05:18.687410 6462 ts_tablet_manager.cc:1397] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:05:18.687490 6464 tablet_bootstrap.cc:492] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9: No bootstrap required, opened a new log
I20260923 23:05:18.687553 6464 ts_tablet_manager.cc:1397] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:05:18.689522 6462 raft_consensus.cc:348] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } }
I20260923 23:05:18.689569 6464 raft_consensus.cc:348] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } }
I20260923 23:05:18.689574 6463 raft_consensus.cc:348] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } }
I20260923 23:05:18.689702 6462 raft_consensus.cc:374] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:18.689726 6463 raft_consensus.cc:374] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:18.689726 6464 raft_consensus.cc:374] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:18.689749 6462 raft_consensus.cc:729] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6954cb9d32743b185b740f00781c57c, State: Initialized, Role: FOLLOWER
I20260923 23:05:18.689761 6463 raft_consensus.cc:729] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4025f5800d42469b081c6d57d6f50c, State: Initialized, Role: FOLLOWER
I20260923 23:05:18.689761 6464 raft_consensus.cc:729] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f34915ac8d0456f809b797116b656b9, State: Initialized, Role: FOLLOWER
I20260923 23:05:18.689867 6463 consensus_queue.cc:260] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } }
I20260923 23:05:18.689862 6462 consensus_queue.cc:260] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } }
I20260923 23:05:18.689886 6464 consensus_queue.cc:260] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } }
I20260923 23:05:18.690156 6464 ts_tablet_manager.cc:1428] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:05:18.690156 6463 ts_tablet_manager.cc:1428] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:05:18.690171 6462 ts_tablet_manager.cc:1428] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 23:05:18.690171 6177 heartbeater.cc:503] Master 127.5.74.62:35175 was elected leader, sending a full tablet report...
I20260923 23:05:18.690276 6445 heartbeater.cc:503] Master 127.5.74.62:35175 was elected leader, sending a full tablet report...
I20260923 23:05:18.690362 6311 heartbeater.cc:503] Master 127.5.74.62:35175 was elected leader, sending a full tablet report...
W20260923 23:05:18.712886 6312 tablet.cc:2369] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:18.782019 6468 raft_consensus.cc:482] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:18.782155 6468 raft_consensus.cc:504] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } }
I20260923 23:05:18.782649 6468 leader_election.cc:288] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6954cb9d32743b185b740f00781c57c (127.5.74.1:34755), 6f34915ac8d0456f809b797116b656b9 (127.5.74.3:33261)
I20260923 23:05:18.786125 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.783040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:50743 (local address 127.5.74.3:33261)
0923 23:05:18.783086 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.783088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.783181 (+ 93us) server_negotiation.cc:407] Connection header received
0923 23:05:18.783250 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.783251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.783263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.783326 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:18.783562 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.784062 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.785310 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.785772 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.785828 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.785838 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.785994 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.786025 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:05:18.786123 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.782900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:33261 (local address 127.5.74.2:50743)
0923 23:05:18.783098 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:18.783108 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:18.783174 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:18.783384 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:18.783384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:18.783386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:18.783479 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.783480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.784190 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.784191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.785224 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.785226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.785845 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.785846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.785920 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.785925 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:18.785983 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:05:18.786024 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":84,"thread_start_us":116,"threads_started":1}
I20260923 23:05:18.786213 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.783060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:48245 (local address 127.5.74.1:34755)
0923 23:05:18.783106 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.783108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.783120 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:05:18.783150 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.783151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.783164 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.783239 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:18.783480 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.784051 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.785310 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.785827 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.785903 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.785916 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.786100 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.786144 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:05:18.786250 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.782900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:34755 (local address 127.5.74.2:48245)
0923 23:05:18.782985 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:18.782999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:18.783068 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:18.783249 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:18.783250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:18.783251 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:18.783330 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.783333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.784134 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.784135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.785220 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:18.785223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.785943 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.785944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:18.786023 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.786025 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:05:18.786076 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:05:18.786108 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":20}
I20260923 23:05:18.786495 6399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e51fc5429a047f091f9648a9edaecd0" candidate_uuid: "ff4025f5800d42469b081c6d57d6f50c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f34915ac8d0456f809b797116b656b9" is_pre_election: true
I20260923 23:05:18.786552 6131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e51fc5429a047f091f9648a9edaecd0" candidate_uuid: "ff4025f5800d42469b081c6d57d6f50c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6954cb9d32743b185b740f00781c57c" is_pre_election: true
I20260923 23:05:18.786662 6399 raft_consensus.cc:2454] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4025f5800d42469b081c6d57d6f50c in term 0.
I20260923 23:05:18.786703 6131 raft_consensus.cc:2454] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4025f5800d42469b081c6d57d6f50c in term 0.
I20260923 23:05:18.786923 6198 leader_election.cc:302] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f34915ac8d0456f809b797116b656b9, ff4025f5800d42469b081c6d57d6f50c; no voters:
I20260923 23:05:18.787062 6468 raft_consensus.cc:2781] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:18.787115 6468 raft_consensus.cc:482] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:18.787180 6468 raft_consensus.cc:3037] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 0 FOLLOWER]: Advancing to term 1
W20260923 23:05:18.787281 6178 tablet.cc:2369] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:18.787992 6468 raft_consensus.cc:504] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } }
I20260923 23:05:18.788141 6468 leader_election.cc:288] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [CANDIDATE]: Term 1 election: Requested vote from peers e6954cb9d32743b185b740f00781c57c (127.5.74.1:34755), 6f34915ac8d0456f809b797116b656b9 (127.5.74.3:33261)
I20260923 23:05:18.788313 6399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e51fc5429a047f091f9648a9edaecd0" candidate_uuid: "ff4025f5800d42469b081c6d57d6f50c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f34915ac8d0456f809b797116b656b9"
I20260923 23:05:18.788329 6131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e51fc5429a047f091f9648a9edaecd0" candidate_uuid: "ff4025f5800d42469b081c6d57d6f50c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6954cb9d32743b185b740f00781c57c"
I20260923 23:05:18.788395 6399 raft_consensus.cc:3037] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:18.788404 6131 raft_consensus.cc:3037] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:18.789216 6399 raft_consensus.cc:2454] T 0e51fc5429a047f091f9648a9edaecd0 P 6f34915ac8d0456f809b797116b656b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4025f5800d42469b081c6d57d6f50c in term 1.
I20260923 23:05:18.789250 6131 raft_consensus.cc:2454] T 0e51fc5429a047f091f9648a9edaecd0 P e6954cb9d32743b185b740f00781c57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4025f5800d42469b081c6d57d6f50c in term 1.
I20260923 23:05:18.789443 6200 leader_election.cc:302] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6954cb9d32743b185b740f00781c57c, ff4025f5800d42469b081c6d57d6f50c; no voters:
I20260923 23:05:18.789552 6468 raft_consensus.cc:2781] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:18.789696 6468 raft_consensus.cc:686] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [term 1 LEADER]: Becoming Leader. State: Replica: ff4025f5800d42469b081c6d57d6f50c, State: Running, Role: LEADER
I20260923 23:05:18.789801 6468 consensus_queue.cc:237] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } }
I20260923 23:05:18.790751 5986 catalog_manager.cc:5495] T 0e51fc5429a047f091f9648a9edaecd0 P ff4025f5800d42469b081c6d57d6f50c reported cstate change: term changed from 0 to 1, leader changed from <none> to ff4025f5800d42469b081c6d57d6f50c (127.5.74.2). New cstate: current_term: 1 leader_uuid: "ff4025f5800d42469b081c6d57d6f50c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6954cb9d32743b185b740f00781c57c" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff4025f5800d42469b081c6d57d6f50c" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f34915ac8d0456f809b797116b656b9" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33261 } health_report { overall_health: UNKNOWN } } }
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:18.805666 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.796962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.5.74.1:34755)
0923 23:05:18.797085 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.797087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.797179 (+ 92us) server_negotiation.cc:407] Connection header received
0923 23:05:18.797220 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.797221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.797231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.797282 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:18.797930 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.798406 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.799292 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.799590 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.799671 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.802397 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:18.802410 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:18.802411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:18.802419 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:18.804412 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.804819 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.804822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.804823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.804891 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.805103 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.805104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.805105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.805249 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:18.805365 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.805490 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.805554 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:18.830785 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.822601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.5.74.1:34755)
0923 23:05:18.822697 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.822699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.822796 (+ 97us) server_negotiation.cc:407] Connection header received
0923 23:05:18.822839 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.822841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.822851 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.822921 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:18.823623 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.824110 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.824968 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.825158 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.825225 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.827911 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:18.827913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:18.827913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:18.827920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:18.829391 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.829782 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.829784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.829784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.829844 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.830025 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.830026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.830026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.830134 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:18.830220 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.830521 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.830597 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:18.856662 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:18.849147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.5.74.1:34755)
0923 23:05:18.849214 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:18.849216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:18.849332 (+ 116us) server_negotiation.cc:407] Connection header received
0923 23:05:18.849421 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:18.849421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:18.849461 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:18.849530 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:18.850133 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.850611 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.851379 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:18.851570 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:18.851659 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:18.854030 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:18.854032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:18.854033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:18.854041 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:18.855466 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.855843 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.855845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.855846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.855903 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:18.856121 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:18.856122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:18.856122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:18.856259 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:18.856367 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:18.856470 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 23:05:18.856547 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260923 23:05:18.857234 6091 tablet_service.cc:2908] Rejecting scan request for tablet 0e51fc5429a047f091f9648a9edaecd0: Uninitialized: clean time has not yet been initialized
I20260923 23:05:18.858269 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 6048
I20260923 23:05:18.863647 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 6182
I20260923 23:05:18.868821 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 6316
I20260923 23:05:18.873535 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 5954
2026-09-23T23:05:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2169 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[6486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[6486](info): set up 2 sockets
Sep 23 23:05:18 dist-test-slave-j4c5 krb5kdc[6486](info): commencing operation
krb5kdc: starting...
W20260923 23:05:19.944104 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:20.085135 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41093
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:20.186188 6499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:20.186468 6499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:20.186497 6499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:20.188074 6499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:20.188314 6499 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 23:05:20.188345 6499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:20.188362 6499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:20.188376 6499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:20.188390 6499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:20.191030 6499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:41093
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:20.191359 6499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:20.191706 6499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:20.200039 6505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:20.200052 6504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:20.200444 6499 server_base.cc:1034] running on GCE node
W20260923 23:05:20.200551 6507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:20.200937 6499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:20.201298 6499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:20.202545 6499 hybrid_clock.cc:648] HybridClock initialized: now 1790204720202514 us; error 46 us; skew 500 ppm
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:20.206421 6499 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:20.207206 6499 webserver.cc:460] Webserver started at http://127.5.74.62:43325/ using document root <none> and password file <none>
I20260923 23:05:20.207535 6499 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:20.207600 6499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:20.207742 6499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:20.210397 6499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "65a7336ab83a4e8bb4f8f9762759ca27"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "7497706884e51e487c60000acf4e557a"
server_key_iv: "12c3bd16dc63d7386a09cdf6d4e28fd1"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.210999 6499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "65a7336ab83a4e8bb4f8f9762759ca27"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "7497706884e51e487c60000acf4e557a"
server_key_iv: "12c3bd16dc63d7386a09cdf6d4e28fd1"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.213414 6499 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260923 23:05:20.214560 6514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.214782 6499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:20.214892 6499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "65a7336ab83a4e8bb4f8f9762759ca27"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "7497706884e51e487c60000acf4e557a"
server_key_iv: "12c3bd16dc63d7386a09cdf6d4e28fd1"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.215042 6499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:20.228261 6499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:20.233860 6499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:20.234081 6499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:20.240887 6499 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:41093
I20260923 23:05:20.240970 6576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:41093 every 8 connection(s)
I20260923 23:05:20.241427 6499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:20.242395 6577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:20.242547 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 6499
I20260923 23:05:20.242646 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:20.242980 5416 external_mini_cluster.cc:1428] Setting key 5ebd5a42aecf3462564a2a20e5647f50
I20260923 23:05:20.247128 6577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27: Bootstrap starting.
I20260923 23:05:20.248255 6577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:20.248631 6577 log.cc:824] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:20.249604 6577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27: No bootstrap required, opened a new log
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:20.253368 6577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a7336ab83a4e8bb4f8f9762759ca27" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41093 } }
I20260923 23:05:20.253633 6577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:20.253683 6577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a7336ab83a4e8bb4f8f9762759ca27, State: Initialized, Role: FOLLOWER
I20260923 23:05:20.253881 6577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a7336ab83a4e8bb4f8f9762759ca27" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41093 } }
I20260923 23:05:20.253986 6577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:20.254029 6577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:20.254084 6577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:20.254936 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.243618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.5.74.62:41093)
0923 23:05:20.243803 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.243813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.243858 (+ 45us) server_negotiation.cc:407] Connection header received
0923 23:05:20.244492 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.244500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.244587 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.244737 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:20.245396 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.246420 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.247374 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.247865 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.248004 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.250879 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:20.250885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:20.250886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:20.250908 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:20.252955 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.253404 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.253408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.253409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.253518 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.253766 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.253767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.253767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.254139 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:20.254267 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.254534 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.254690 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260923 23:05:20.255040 6577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a7336ab83a4e8bb4f8f9762759ca27" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41093 } }
I20260923 23:05:20.255154 6577 leader_election.cc:302] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a7336ab83a4e8bb4f8f9762759ca27; no voters:
I20260923 23:05:20.255394 6577 leader_election.cc:288] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:20.255514 6582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:20.255673 6577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:20.255787 6582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [term 1 LEADER]: Becoming Leader. State: Replica: 65a7336ab83a4e8bb4f8f9762759ca27, State: Running, Role: LEADER
I20260923 23:05:20.255939 6582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a7336ab83a4e8bb4f8f9762759ca27" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41093 } }
I20260923 23:05:20.256466 6584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a7336ab83a4e8bb4f8f9762759ca27. Latest consensus state: current_term: 1 leader_uuid: "65a7336ab83a4e8bb4f8f9762759ca27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a7336ab83a4e8bb4f8f9762759ca27" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41093 } } }
I20260923 23:05:20.256446 6583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65a7336ab83a4e8bb4f8f9762759ca27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a7336ab83a4e8bb4f8f9762759ca27" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41093 } } }
I20260923 23:05:20.256584 6584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:20.256590 6583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27 [sys.catalog]: This master's current role is: LEADER
W20260923 23:05:20.257747 6598 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:05:20.257819 6598 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:05:20.257916 6591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:20.258203 6591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:20.261821 6591 catalog_manager.cc:1347] Generated new cluster ID: 3430d994b5b34d69bfbf99b026b6b893
I20260923 23:05:20.261921 6591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:20.285969 6591 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:20.286738 6591 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:20.291930 6591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27: Generated new TSK 0
I20260923 23:05:20.292192 6591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:20.324434 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41093
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:20.415930 6603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:20.416455 6603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:20.416556 6603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:20.418248 6603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:20.418375 6603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:20.418401 6603 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:05:20.418421 6603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:20.420395 6603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.74.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:20.420691 6603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:20.420948 6603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:20.426498 6611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:20.426502 6609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:20.426498 6608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:20.426528 6603 server_base.cc:1034] running on GCE node
I20260923 23:05:20.427052 6603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:20.427378 6603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:20.428570 6603 hybrid_clock.cc:648] HybridClock initialized: now 1790204720428553 us; error 34 us; skew 500 ppm
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:20.431705 6603 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:20.432353 6603 webserver.cc:460] Webserver started at http://127.5.74.1:44967/ using document root <none> and password file <none>
I20260923 23:05:20.432628 6603 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:20.432705 6603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:20.432875 6603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:20.434068 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "74a58fea6c6f491ba22cd39197a36981"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "451fa57b511a459dab638ea2659b562c"
server_key_iv: "36f6cc6b5ea8c45ed93d7fd7ab048269"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.434546 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "74a58fea6c6f491ba22cd39197a36981"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "451fa57b511a459dab638ea2659b562c"
server_key_iv: "36f6cc6b5ea8c45ed93d7fd7ab048269"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.436086 6603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:20.437132 6618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.437352 6603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:20.437469 6603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "74a58fea6c6f491ba22cd39197a36981"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "451fa57b511a459dab638ea2659b562c"
server_key_iv: "36f6cc6b5ea8c45ed93d7fd7ab048269"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.437558 6603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:20.448470 6603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:20.450730 6603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:20.450901 6603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:20.451445 6603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:20.451511 6603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.451560 6603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:20.451594 6603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.458662 6603 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:35523
I20260923 23:05:20.458724 6733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:35523 every 8 connection(s)
I20260923 23:05:20.459074 6603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:20.460237 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 6603
I20260923 23:05:20.460337 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:20.460536 5416 external_mini_cluster.cc:1428] Setting key 6f358f517b306fb78149a4884fb17c06
I20260923 23:05:20.462826 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.452066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:33395 (local address 127.5.74.62:41093)
0923 23:05:20.452116 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.452118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.452387 (+ 269us) server_negotiation.cc:407] Connection header received
0923 23:05:20.452957 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.452959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.452973 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.453046 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:20.453989 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.454858 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.456482 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.456894 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.456969 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.459745 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:20.459747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:20.459747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:20.459754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:20.461435 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.461862 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.461863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.461864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.461923 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.462174 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.462175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.462175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.462298 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:20.462376 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.462640 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.462725 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:05:20.463027 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.452031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41093 (local address 127.5.74.1:33395)
0923 23:05:20.452247 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.452299 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.452811 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.453138 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.453143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.453586 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:20.453874 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.453882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.454955 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.454956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.456396 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.456399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.457010 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.457011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.457180 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.457896 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:20.457908 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:20.459659 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:20.461534 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.461542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.461544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.461764 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.461996 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.461997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.461997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.462103 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.462366 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:20.462371 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:20.462509 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.462621 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.462784 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":127,"threads_started":1}
I20260923 23:05:20.463537 6734 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41093
I20260923 23:05:20.463670 6734 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:20.463884 6734 heartbeater.cc:511] Master 127.5.74.62:41093 requested a full tablet report, sending...
I20260923 23:05:20.464428 6531 ts_manager.cc:194] Registered new tserver with Master: 74a58fea6c6f491ba22cd39197a36981 (127.5.74.1:35523)
I20260923 23:05:20.465417 6531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:33395
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:05:20.472118 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.464768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:48195 (local address 127.5.74.62:41093)
0923 23:05:20.464906 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.464908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.464915 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:05:20.464933 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.464934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.464941 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.464990 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:20.465324 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.466111 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.467370 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.467601 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.467679 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.468887 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:20.468889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:20.468889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:20.468897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:20.470713 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.471076 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.471077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.471077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.471152 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.471587 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.471588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.471589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.471721 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:20.471799 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.471989 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.472055 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260923 23:05:20.472280 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.464703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41093 (local address 127.5.74.1:48195)
0923 23:05:20.464839 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.464850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.464894 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.464995 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.464996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.465187 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:20.465246 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.465247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.466232 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.466233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.467264 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.467275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.467700 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.467701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.467779 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.468407 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:20.468413 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:20.468757 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:20.470802 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.470803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.470804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.470991 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.471247 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.471248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.471249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.471381 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.471844 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:20.471845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:20.471887 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.471962 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.472049 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:20.492805 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41093
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:20.577900 6743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:20.578123 6743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:20.578152 6743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:20.579308 6743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:20.579345 6743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:20.579366 6743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:05:20.579384 6743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:20.580814 6743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.74.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:20.581014 6743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:20.581202 6743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:20.587646 6748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:20.587622 6749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:20.587677 6743 server_base.cc:1034] running on GCE node
W20260923 23:05:20.587622 6751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:20.588356 6743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:20.588598 6743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:20.589809 6743 hybrid_clock.cc:648] HybridClock initialized: now 1790204720589757 us; error 47 us; skew 500 ppm
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:20.593364 6743 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:20.593962 6743 webserver.cc:460] Webserver started at http://127.5.74.2:38743/ using document root <none> and password file <none>
I20260923 23:05:20.594213 6743 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:20.594304 6743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:20.594458 6743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:20.595651 6743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "a7aeb2dd48ef46db8cca98c550ac889e"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "cdb5f5cf074a7e4c841f752764ddc0fc"
server_key_iv: "2c54e91910adc0e985d0a7b2437a61a1"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.596102 6743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "a7aeb2dd48ef46db8cca98c550ac889e"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "cdb5f5cf074a7e4c841f752764ddc0fc"
server_key_iv: "2c54e91910adc0e985d0a7b2437a61a1"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.597805 6743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:20.599023 6758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.599418 6743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:20.599555 6743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "a7aeb2dd48ef46db8cca98c550ac889e"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "cdb5f5cf074a7e4c841f752764ddc0fc"
server_key_iv: "2c54e91910adc0e985d0a7b2437a61a1"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.599645 6743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:20.610591 6743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:20.613179 6743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:20.613364 6743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:20.613920 6743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:20.613984 6743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.614037 6743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:20.614068 6743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:20.622262 6743 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:35987
I20260923 23:05:20.622381 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:35987 every 8 connection(s)
I20260923 23:05:20.622710 6743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:05:20.625538 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.614611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:33151 (local address 127.5.74.62:41093)
0923 23:05:20.614663 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.614666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.614897 (+ 231us) server_negotiation.cc:407] Connection header received
0923 23:05:20.615473 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.615474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.615487 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.615551 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:20.616493 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.616994 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.618202 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.618423 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.618512 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.622097 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:20.622099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:20.622099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:20.622109 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:20.623940 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.624473 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.624475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.624475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.624539 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.624841 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.624842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.624842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.624970 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:20.625050 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.625343 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.625434 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 23:05:20.625725 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.614568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41093 (local address 127.5.74.2:33151)
0923 23:05:20.614735 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.614802 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.615360 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.615647 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.615652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.616113 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:20.616386 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.616391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.617089 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.617090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.618113 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.618115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.618524 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.618525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.618655 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.619346 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:20.619356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:20.621958 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:20.624057 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.624064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.624066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.624383 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.624610 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.624611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.624611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.624732 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.625039 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:20.625046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:20.625199 (+ 153us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.625307 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.625477 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260923 23:05:20.626194 6874 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41093
I20260923 23:05:20.626281 6874 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:20.626482 6874 heartbeater.cc:511] Master 127.5.74.62:41093 requested a full tablet report, sending...
I20260923 23:05:20.626964 6530 ts_manager.cc:194] Registered new tserver with Master: a7aeb2dd48ef46db8cca98c550ac889e (127.5.74.2:35987)
I20260923 23:05:20.627485 6530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:33151
I20260923 23:05:20.628203 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 6743
I20260923 23:05:20.628324 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:20.628492 5416 external_mini_cluster.cc:1428] Setting key e79fdfe52d605466ae355f0d4ef7ead6
I20260923 23:05:20.634847 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.627323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41093 (local address 127.5.74.2:42953)
0923 23:05:20.627714 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.627725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.627847 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.628633 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.628634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.628869 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:20.628943 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.628945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.629698 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.629699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.630234 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.630236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.630625 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.630625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.630691 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.631299 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:20.631350 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:20.631697 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:20.633690 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.633691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.633692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.633881 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.634098 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.634099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.634100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.634161 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.634471 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:20.634472 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:20.634530 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.634610 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.634730 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":100,"threads_started":1}
I20260923 23:05:20.634840 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.627405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:42953 (local address 127.5.74.62:41093)
0923 23:05:20.627446 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.627448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.627829 (+ 381us) server_negotiation.cc:407] Connection header received
0923 23:05:20.627908 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.627909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.627919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.627997 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:20.629039 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.629609 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.630344 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.630540 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.630624 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.631865 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:20.631866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:20.631867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:20.631873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:20.633581 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.633969 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.633970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.633971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.634029 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.634233 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.634234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.634235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.634390 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:20.634467 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.634650 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.634728 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:20.660254 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41093
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:20.749212 6883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:20.749569 6883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:20.749612 6883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:20.750919 6883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:20.751003 6883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:20.751039 6883 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:05:20.751078 6883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:20.752915 6883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.74.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:20.753221 6883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:20.753476 6883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:20.759249 6888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:20.759254 6889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:20.759298 6891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:20.759546 6883 server_base.cc:1034] running on GCE node
I20260923 23:05:20.759780 6883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:20.760062 6883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:20.761277 6883 hybrid_clock.cc:648] HybridClock initialized: now 1790204720761272 us; error 42 us; skew 500 ppm
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:20.764739 6883 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:20.765280 6883 webserver.cc:460] Webserver started at http://127.5.74.3:33279/ using document root <none> and password file <none>
I20260923 23:05:20.765539 6883 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:20.765610 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:20.765784 6883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:20.766954 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "0e299f1ab1914a55b731eca71c676695"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "0663bf552cd28e0db86c4be14f072b50"
server_key_iv: "fa9a7dd5533fb935f7aafc269e6ba2e3"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.767464 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "0e299f1ab1914a55b731eca71c676695"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "0663bf552cd28e0db86c4be14f072b50"
server_key_iv: "fa9a7dd5533fb935f7aafc269e6ba2e3"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.769119 6883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260923 23:05:20.770124 6898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.770741 6883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:20.770908 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "0e299f1ab1914a55b731eca71c676695"
format_stamp: "Formatted at 2026-09-23 23:05:20 on dist-test-slave-j4c5"
server_key: "0663bf552cd28e0db86c4be14f072b50"
server_key_iv: "fa9a7dd5533fb935f7aafc269e6ba2e3"
server_key_version: "encryptionkey@0"
I20260923 23:05:20.771000 6883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:20.778806 6883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:20.781224 6883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:20.781404 6883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:20.782030 6883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:20.782088 6883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.782122 6883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:20.782145 6883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:20.789738 6883 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:34909
I20260923 23:05:20.789893 7013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:34909 every 8 connection(s)
I20260923 23:05:20.790207 6883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:05:20.793030 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.782511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41093 (local address 127.5.74.3:53829)
0923 23:05:20.782663 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.782703 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.783229 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.783498 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.783503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.783889 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:20.784237 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.784244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.785274 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.785276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.786105 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.786107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.786562 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.786563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.786725 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.787342 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:20.787351 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:20.789448 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:20.791377 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.791382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.791384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.791682 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.791990 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.791991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.791992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.792139 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.792505 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:20.792510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:20.792635 (+ 125us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.792719 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.792864 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260923 23:05:20.793037 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.782472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:53829 (local address 127.5.74.62:41093)
0923 23:05:20.782543 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.782546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.782779 (+ 233us) server_negotiation.cc:407] Connection header received
0923 23:05:20.783347 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.783349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.783361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.783408 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:20.784582 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.785159 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.786266 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.786463 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.786545 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.789585 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:20.789587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:20.789588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:20.789597 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:20.791275 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.791816 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.791817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.791817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.791900 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.792262 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.792263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.792263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.792398 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:20.792506 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.792887 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.792974 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:05:20.793589 7014 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41093
I20260923 23:05:20.793676 7014 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:20.793871 7014 heartbeater.cc:511] Master 127.5.74.62:41093 requested a full tablet report, sending...
I20260923 23:05:20.794497 6531 ts_manager.cc:194] Registered new tserver with Master: 0e299f1ab1914a55b731eca71c676695 (127.5.74.3:34909)
I20260923 23:05:20.794981 6531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:53829
I20260923 23:05:20.796490 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 6883
I20260923 23:05:20.796579 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:20.796727 5416 external_mini_cluster.cc:1428] Setting key 2c49957f06f8a427924661cb652d017a
I20260923 23:05:20.797753 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:05:20.802083 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.794769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41093 (local address 127.5.74.3:51781)
0923 23:05:20.794904 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.794915 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.794978 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.795369 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.795370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.795670 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:20.795729 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.795731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.796843 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.796844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.797753 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.797756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.798215 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.798215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.798272 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.798761 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:20.798766 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:20.799086 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:20.800964 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.800965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.800966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.801159 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.801409 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:20.801410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:20.801410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:20.801506 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:20.801807 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:20.801808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:20.801850 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.801906 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.802001 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260923 23:05:20.802177 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.794863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:51781 (local address 127.5.74.62:41093)
0923 23:05:20.794901 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.794903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.794964 (+ 61us) server_negotiation.cc:407] Connection header received
0923 23:05:20.795033 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.795034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.795060 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.795222 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:20.795868 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.796744 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.797868 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.798136 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.798210 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.799188 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:20.799190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:20.799190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:20.799199 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:20.800858 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.801252 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.801253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.801253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.801315 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.801584 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.801585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.801585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.801740 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:20.801810 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.801950 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.802037 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:05:20 dist-test-slave-j4c5 krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:20.835079 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.824696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.5.74.62:41093)
0923 23:05:20.824786 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.824789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.824802 (+ 13us) server_negotiation.cc:407] Connection header received
0923 23:05:20.824839 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.824840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.824855 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.824992 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:20.826070 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.826855 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.828393 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.828763 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.828872 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.832332 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:20.832334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:20.832334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:20.832342 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:20.833625 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.834115 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.834117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.834117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.834191 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:20.834498 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:20.834498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:20.834499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:20.834646 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:20.834736 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.834923 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.835001 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 23:05:20.837405 6531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:20.839003 6531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:20.845867 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.841739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:34909 (local address 127.0.0.1:57470)
0923 23:05:20.842129 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.842140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.842214 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.842867 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.842868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.842870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:20.842911 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.842912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.844027 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.844028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.844965 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.844967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.845622 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.845623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.845689 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.845693 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.845745 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.845788 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260923 23:05:20.845887 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.841934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.5.74.2:35987)
0923 23:05:20.842122 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.842128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.842139 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:20.842182 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.842184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.842201 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.842308 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:20.842480 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.843652 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.844948 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.845474 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.845533 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.845601 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.845736 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.845779 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20260923 23:05:20.845875 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.841739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35987 (local address 127.0.0.1:43540)
0923 23:05:20.841930 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.841970 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.842041 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.842325 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.842328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.842330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:20.842402 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.842404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.843739 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.843740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.844849 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.844852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.845559 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.845560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.845635 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.845648 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.845718 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.845774 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":6,"thread_start_us":48,"threads_started":1}
I20260923 23:05:20.846737 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.841981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.5.74.3:34909)
0923 23:05:20.842661 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.842665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.842674 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:20.842708 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.842709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.842724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.842796 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:20.842983 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.843941 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.845038 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.845529 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.845602 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.845660 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.846590 (+ 930us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.846636 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":74,"threads_started":1}
I20260923 23:05:20.846764 6808 tablet_service.cc:1467] Processing CreateTablet for tablet ab41cd0a70e3463b991efe97daafdef7 (DEFAULT_TABLE table=test-table [id=d2f91ac4166c4e23bb3ba33d9bdd807d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:20.847267 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.841747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35523 (local address 127.0.0.1:58210)
0923 23:05:20.842149 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.842159 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.842201 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.842469 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.842470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.842473 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:20.842527 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.842529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.843470 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.843471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.844849 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.844852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.847036 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.847037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.847104 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.847109 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.847154 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.847199 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":91,"threads_started":1}
I20260923 23:05:20.847278 6808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab41cd0a70e3463b991efe97daafdef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:20.847496 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.841970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.5.74.1:35523)
0923 23:05:20.842146 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.842151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.842264 (+ 113us) server_negotiation.cc:407] Connection header received
0923 23:05:20.842305 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.842307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.842321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.842385 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:20.842630 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.843377 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.845638 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.846717 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.847034 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.847150 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.847177 (+ 27us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.847216 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260923 23:05:20.847429 6948 tablet_service.cc:1467] Processing CreateTablet for tablet ab41cd0a70e3463b991efe97daafdef7 (DEFAULT_TABLE table=test-table [id=d2f91ac4166c4e23bb3ba33d9bdd807d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:20.847750 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab41cd0a70e3463b991efe97daafdef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:20.848053 6668 tablet_service.cc:1467] Processing CreateTablet for tablet ab41cd0a70e3463b991efe97daafdef7 (DEFAULT_TABLE table=test-table [id=d2f91ac4166c4e23bb3ba33d9bdd807d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:20.848381 6668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab41cd0a70e3463b991efe97daafdef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:20.851130 7037 tablet_bootstrap.cc:492] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: Bootstrap starting.
I20260923 23:05:20.851853 7037 tablet_bootstrap.cc:654] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:20.852200 7037 log.cc:824] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:20.852325 7038 tablet_bootstrap.cc:492] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: Bootstrap starting.
I20260923 23:05:20.852396 7039 tablet_bootstrap.cc:492] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: Bootstrap starting.
I20260923 23:05:20.853008 7039 tablet_bootstrap.cc:654] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:20.853058 7037 tablet_bootstrap.cc:492] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: No bootstrap required, opened a new log
I20260923 23:05:20.853119 7037 ts_tablet_manager.cc:1397] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:05:20.853154 7038 tablet_bootstrap.cc:654] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:20.853369 7038 log.cc:824] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:20.853322 7039 log.cc:824] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:20.854094 7038 tablet_bootstrap.cc:492] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: No bootstrap required, opened a new log
I20260923 23:05:20.854167 7038 ts_tablet_manager.cc:1397] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:20.855499 7039 tablet_bootstrap.cc:492] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: No bootstrap required, opened a new log
I20260923 23:05:20.855581 7039 ts_tablet_manager.cc:1397] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:05:20.855481 7037 raft_consensus.cc:348] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } }
I20260923 23:05:20.855679 7037 raft_consensus.cc:374] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:20.855723 7037 raft_consensus.cc:729] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a58fea6c6f491ba22cd39197a36981, State: Initialized, Role: FOLLOWER
I20260923 23:05:20.855918 7037 consensus_queue.cc:260] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } }
I20260923 23:05:20.856315 7037 ts_tablet_manager.cc:1428] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260923 23:05:20.856463 6734 heartbeater.cc:503] Master 127.5.74.62:41093 was elected leader, sending a full tablet report...
I20260923 23:05:20.856886 7038 raft_consensus.cc:348] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } }
I20260923 23:05:20.857020 7038 raft_consensus.cc:374] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:20.857046 7038 raft_consensus.cc:729] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e299f1ab1914a55b731eca71c676695, State: Initialized, Role: FOLLOWER
I20260923 23:05:20.857180 7038 consensus_queue.cc:260] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } }
I20260923 23:05:20.857499 7038 ts_tablet_manager.cc:1428] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:05:20.857633 7014 heartbeater.cc:503] Master 127.5.74.62:41093 was elected leader, sending a full tablet report...
I20260923 23:05:20.857859 7039 raft_consensus.cc:348] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } }
I20260923 23:05:20.858018 7039 raft_consensus.cc:374] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:20.858059 7039 raft_consensus.cc:729] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7aeb2dd48ef46db8cca98c550ac889e, State: Initialized, Role: FOLLOWER
I20260923 23:05:20.858167 7039 consensus_queue.cc:260] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } }
I20260923 23:05:20.858433 7039 ts_tablet_manager.cc:1428] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:05:20.858507 6874 heartbeater.cc:503] Master 127.5.74.62:41093 was elected leader, sending a full tablet report...
W20260923 23:05:20.873862 6875 tablet.cc:2369] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:20.890618 7045 raft_consensus.cc:482] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:20.890751 7045 raft_consensus.cc:504] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } }
I20260923 23:05:20.891124 7045 leader_election.cc:288] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e299f1ab1914a55b731eca71c676695 (127.5.74.3:34909), 74a58fea6c6f491ba22cd39197a36981 (127.5.74.1:35523)
I20260923 23:05:20.894356 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.891424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:34909 (local address 127.5.74.2:40059)
0923 23:05:20.891613 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.891621 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.891695 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.891822 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.891822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.891824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:20.891888 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.891890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.892603 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.892604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.893472 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.893474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.894060 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.894061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.894136 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.894140 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.894218 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.894263 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":135,"threads_started":1}
I20260923 23:05:20.894340 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.891505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:40059 (local address 127.5.74.3:34909)
0923 23:05:20.891560 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.891563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.891693 (+ 130us) server_negotiation.cc:407] Connection header received
0923 23:05:20.891745 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.891746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.891757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.891816 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:20.891966 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.892518 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.893562 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.893973 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.894058 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.894069 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.894223 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.894263 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 23:05:20.894608 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.891388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35523 (local address 127.5.74.2:54239)
0923 23:05:20.891471 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.891483 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.891535 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.891806 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.891808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.891810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:20.891899 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.891901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.892693 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.892694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.893600 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.893602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.894245 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.894246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.894347 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.894351 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.894435 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.894497 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 23:05:20.894703 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.891466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:54239 (local address 127.5.74.1:35523)
0923 23:05:20.891509 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.891511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.891519 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:20.891636 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.891638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.891658 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.891797 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:20.891989 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.892594 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.893720 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.894136 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.894241 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.894252 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.894590 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.894640 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:05:20.894729 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab41cd0a70e3463b991efe97daafdef7" candidate_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e299f1ab1914a55b731eca71c676695" is_pre_election: true
I20260923 23:05:20.894915 6968 raft_consensus.cc:2454] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7aeb2dd48ef46db8cca98c550ac889e in term 0.
I20260923 23:05:20.894936 6688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab41cd0a70e3463b991efe97daafdef7" candidate_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a58fea6c6f491ba22cd39197a36981" is_pre_election: true
I20260923 23:05:20.895069 6688 raft_consensus.cc:2454] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7aeb2dd48ef46db8cca98c550ac889e in term 0.
I20260923 23:05:20.895164 6760 leader_election.cc:302] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e299f1ab1914a55b731eca71c676695, a7aeb2dd48ef46db8cca98c550ac889e; no voters:
I20260923 23:05:20.895370 7045 raft_consensus.cc:2781] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:20.895455 7045 raft_consensus.cc:482] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:20.895491 7045 raft_consensus.cc:3037] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:20.896389 7045 raft_consensus.cc:504] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } }
I20260923 23:05:20.896555 7045 leader_election.cc:288] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [CANDIDATE]: Term 1 election: Requested vote from peers 0e299f1ab1914a55b731eca71c676695 (127.5.74.3:34909), 74a58fea6c6f491ba22cd39197a36981 (127.5.74.1:35523)
I20260923 23:05:20.896721 6688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab41cd0a70e3463b991efe97daafdef7" candidate_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a58fea6c6f491ba22cd39197a36981"
I20260923 23:05:20.896839 6688 raft_consensus.cc:3037] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:20.896946 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab41cd0a70e3463b991efe97daafdef7" candidate_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e299f1ab1914a55b731eca71c676695"
I20260923 23:05:20.897029 6968 raft_consensus.cc:3037] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:20.897687 6688 raft_consensus.cc:2454] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7aeb2dd48ef46db8cca98c550ac889e in term 1.
I20260923 23:05:20.897841 6968 raft_consensus.cc:2454] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7aeb2dd48ef46db8cca98c550ac889e in term 1.
I20260923 23:05:20.898312 6761 leader_election.cc:302] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a58fea6c6f491ba22cd39197a36981, a7aeb2dd48ef46db8cca98c550ac889e; no voters:
I20260923 23:05:20.898483 7045 raft_consensus.cc:2781] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:20.898662 7045 raft_consensus.cc:686] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 1 LEADER]: Becoming Leader. State: Replica: a7aeb2dd48ef46db8cca98c550ac889e, State: Running, Role: LEADER
I20260923 23:05:20.898785 7045 consensus_queue.cc:237] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } }
I20260923 23:05:20.899677 6530 catalog_manager.cc:5495] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7aeb2dd48ef46db8cca98c550ac889e (127.5.74.2). New cstate: current_term: 1 leader_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:20.907773 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.905048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.5.74.2:35987)
0923 23:05:20.905113 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.905116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.905134 (+ 18us) server_negotiation.cc:407] Connection header received
0923 23:05:20.905169 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.905170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.905190 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.905249 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:20.905457 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.905942 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.906744 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.906962 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.907036 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.907260 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:20.907375 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:20.907576 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.907628 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.907656 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:05:20.909790 6968 raft_consensus.cc:1261] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [term 1 FOLLOWER]: Refusing update from remote peer a7aeb2dd48ef46db8cca98c550ac889e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:20.909878 6688 raft_consensus.cc:1261] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Refusing update from remote peer a7aeb2dd48ef46db8cca98c550ac889e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:20.910177 7047 consensus_queue.cc:1035] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:20.910400 7045 consensus_queue.cc:1035] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:20.913208 7050 mvcc.cc:205] Tried to move back new op lower bound from 7332678536843444224 to 7332678536802017280. Current Snapshot: MvccSnapshot[applied={T|T < 7332678536843444224}]
I20260923 23:05:20.914222 7052 mvcc.cc:205] Tried to move back new op lower bound from 7332678536843444224 to 7332678536802017280. Current Snapshot: MvccSnapshot[applied={T|T < 7332678536843444224 or (T in {7332678536843444224})}]
I20260923 23:05:20.915467 7051 mvcc.cc:205] Tried to move back new op lower bound from 7332678536843444224 to 7332678536802017280. Current Snapshot: MvccSnapshot[applied={T|T < 7332678536843444224 or (T in {7332678536843444224})}]
I20260923 23:05:20.923978 6530 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:20.924135 6530 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:20.928766 6530 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27: Sending DeleteTablet for 3 replicas of tablet ab41cd0a70e3463b991efe97daafdef7
I20260923 23:05:20.929229 6948 tablet_service.cc:1514] Processing DeleteTablet for tablet ab41cd0a70e3463b991efe97daafdef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:20 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:57470
I20260923 23:05:20.929251 6668 tablet_service.cc:1514] Processing DeleteTablet for tablet ab41cd0a70e3463b991efe97daafdef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:20 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:58210
I20260923 23:05:20.929522 6808 tablet_service.cc:1514] Processing DeleteTablet for tablet ab41cd0a70e3463b991efe97daafdef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:20 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:43540
I20260923 23:05:20.929579 7061 tablet_replica.cc:330] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: stopping tablet replica
I20260923 23:05:20.929698 7061 raft_consensus.cc:2229] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:20.929646 6530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:20.929816 6530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:20.929836 7061 raft_consensus.cc:2258] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:20.930070 7060 tablet_replica.cc:330] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: stopping tablet replica
I20260923 23:05:20.930135 7062 tablet_replica.cc:330] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: stopping tablet replica
I20260923 23:05:20.930254 7062 raft_consensus.cc:2229] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:20.930332 7060 raft_consensus.cc:2229] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:20.930809 7061 ts_tablet_manager.cc:1905] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:20.930851 7062 raft_consensus.cc:2258] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:20.931325 7062 ts_tablet_manager.cc:1905] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:20.931576 6668 tablet_service.cc:1467] Processing CreateTablet for tablet f96be3cecfc64a49a3c6f039ceed2a2b (DEFAULT_TABLE table=test-table [id=7abc73b8b216400f8b3340b6c0067c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:20.931694 6668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f96be3cecfc64a49a3c6f039ceed2a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:20.933513 7037 tablet_bootstrap.cc:492] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: Bootstrap starting.
I20260923 23:05:20.933513 6808 tablet_service.cc:1467] Processing CreateTablet for tablet f96be3cecfc64a49a3c6f039ceed2a2b (DEFAULT_TABLE table=test-table [id=7abc73b8b216400f8b3340b6c0067c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:20.933741 6808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f96be3cecfc64a49a3c6f039ceed2a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:20.933662 6948 tablet_service.cc:1467] Processing CreateTablet for tablet f96be3cecfc64a49a3c6f039ceed2a2b (DEFAULT_TABLE table=test-table [id=7abc73b8b216400f8b3340b6c0067c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:20.933943 7037 tablet_bootstrap.cc:654] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:20.933964 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f96be3cecfc64a49a3c6f039ceed2a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:20.934317 7061 ts_tablet_manager.cc:1918] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:20.934381 7061 log.cc:1196] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/ab41cd0a70e3463b991efe97daafdef7
I20260923 23:05:20.934643 7061 ts_tablet_manager.cc:1939] T ab41cd0a70e3463b991efe97daafdef7 P 74a58fea6c6f491ba22cd39197a36981: Deleting consensus metadata
I20260923 23:05:20.935082 6517 catalog_manager.cc:4841] TS 74a58fea6c6f491ba22cd39197a36981 (127.5.74.1:35523): tablet ab41cd0a70e3463b991efe97daafdef7 (table test-table [id=d2f91ac4166c4e23bb3ba33d9bdd807d]) successfully deleted
I20260923 23:05:20.935294 7039 tablet_bootstrap.cc:492] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: Bootstrap starting.
I20260923 23:05:20.935678 7039 tablet_bootstrap.cc:654] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:20.935686 7038 tablet_bootstrap.cc:492] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: Bootstrap starting.
I20260923 23:05:20.936216 7060 raft_consensus.cc:2258] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:20.936254 7038 tablet_bootstrap.cc:654] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:20.936420 7062 ts_tablet_manager.cc:1918] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:20.936499 7062 log.cc:1196] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/ab41cd0a70e3463b991efe97daafdef7
I20260923 23:05:20.936754 7062 ts_tablet_manager.cc:1939] T ab41cd0a70e3463b991efe97daafdef7 P a7aeb2dd48ef46db8cca98c550ac889e: Deleting consensus metadata
I20260923 23:05:20.936942 7060 ts_tablet_manager.cc:1905] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:20.937261 6518 catalog_manager.cc:4841] TS a7aeb2dd48ef46db8cca98c550ac889e (127.5.74.2:35987): tablet ab41cd0a70e3463b991efe97daafdef7 (table test-table [id=d2f91ac4166c4e23bb3ba33d9bdd807d]) successfully deleted
I20260923 23:05:20.937479 7039 tablet_bootstrap.cc:492] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: No bootstrap required, opened a new log
I20260923 23:05:20.937518 7039 ts_tablet_manager.cc:1397] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:20.937687 7039 raft_consensus.cc:348] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:20.937748 7039 raft_consensus.cc:374] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:20.937783 7039 raft_consensus.cc:729] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7aeb2dd48ef46db8cca98c550ac889e, State: Initialized, Role: FOLLOWER
I20260923 23:05:20.937842 7039 consensus_queue.cc:260] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:20.937930 7039 ts_tablet_manager.cc:1428] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.939119 7038 tablet_bootstrap.cc:492] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: No bootstrap required, opened a new log
I20260923 23:05:20.939132 7060 ts_tablet_manager.cc:1918] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:20.939234 7038 ts_tablet_manager.cc:1397] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260923 23:05:20.939266 7060 log.cc:1196] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/ab41cd0a70e3463b991efe97daafdef7
I20260923 23:05:20.939451 7038 raft_consensus.cc:348] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:20.939522 7060 ts_tablet_manager.cc:1939] T ab41cd0a70e3463b991efe97daafdef7 P 0e299f1ab1914a55b731eca71c676695: Deleting consensus metadata
I20260923 23:05:20.939539 7038 raft_consensus.cc:374] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:20.939571 7038 raft_consensus.cc:729] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e299f1ab1914a55b731eca71c676695, State: Initialized, Role: FOLLOWER
I20260923 23:05:20.939618 7038 consensus_queue.cc:260] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:20.939709 7038 ts_tablet_manager.cc:1428] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260923 23:05:20.939935 6516 catalog_manager.cc:4841] TS 0e299f1ab1914a55b731eca71c676695 (127.5.74.3:34909): tablet ab41cd0a70e3463b991efe97daafdef7 (table test-table [id=d2f91ac4166c4e23bb3ba33d9bdd807d]) successfully deleted
I20260923 23:05:20.940665 7037 tablet_bootstrap.cc:492] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: No bootstrap required, opened a new log
I20260923 23:05:20.940744 7037 ts_tablet_manager.cc:1397] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.000s
I20260923 23:05:20.940908 7037 raft_consensus.cc:348] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:20.940992 7037 raft_consensus.cc:374] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:20.941028 7037 raft_consensus.cc:729] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a58fea6c6f491ba22cd39197a36981, State: Initialized, Role: FOLLOWER
I20260923 23:05:20.941078 7037 consensus_queue.cc:260] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:20.941195 7037 ts_tablet_manager.cc:1428] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:20.957634 7044 raft_consensus.cc:482] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:20.957788 7044 raft_consensus.cc:504] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:20.958266 7044 leader_election.cc:288] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a58fea6c6f491ba22cd39197a36981 (127.5.74.1:35523), a7aeb2dd48ef46db8cca98c550ac889e (127.5.74.2:35987)
W20260923 23:05:20.959846 6735 tablet.cc:2369] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:20.962795 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.958566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:34475 (local address 127.5.74.2:35987)
0923 23:05:20.958619 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.958621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.958632 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:20.958692 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.958693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.958702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.958768 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:20.958990 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.959924 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.961838 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.962401 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.962497 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.962507 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.962681 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.962725 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:05:20.962821 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.958435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35987 (local address 127.5.74.3:34475)
0923 23:05:20.958550 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.958564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.958622 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.958791 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.958792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.958794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:20.958885 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.958886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.960072 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.960074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.961692 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.961697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.962500 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.962501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.962589 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.962593 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.962662 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.962711 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":98}
I20260923 23:05:20.962838 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.958646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:58015 (local address 127.5.74.1:35523)
0923 23:05:20.958697 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:20.958699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:20.958786 (+ 87us) server_negotiation.cc:407] Connection header received
0923 23:05:20.958887 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:20.958889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:20.958897 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:20.958961 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:20.959185 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.960042 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.961843 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.962420 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.962465 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.962479 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:20.962727 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 23:05:20.962767 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:05:20.963251 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:20.958437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35523 (local address 127.5.74.3:58015)
0923 23:05:20.958724 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:20.958734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:20.958809 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:20.959007 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:20.959008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:20.959010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:20.959095 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.959097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.960136 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.960137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.961692 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:20.961697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:20.962544 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:20.962547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:20.962619 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:20.962622 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:20.963110 (+ 488us) client_negotiation.cc:240] Negotiation successful
0923 23:05:20.963166 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":82,"thread_start_us":65,"threads_started":1}
I20260923 23:05:20.963261 6828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96be3cecfc64a49a3c6f039ceed2a2b" candidate_uuid: "0e299f1ab1914a55b731eca71c676695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" is_pre_election: true
I20260923 23:05:20.963413 6828 raft_consensus.cc:2454] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e299f1ab1914a55b731eca71c676695 in term 0.
I20260923 23:05:20.963531 6688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96be3cecfc64a49a3c6f039ceed2a2b" candidate_uuid: "0e299f1ab1914a55b731eca71c676695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a58fea6c6f491ba22cd39197a36981" is_pre_election: true
I20260923 23:05:20.963624 6688 raft_consensus.cc:2454] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e299f1ab1914a55b731eca71c676695 in term 0.
I20260923 23:05:20.963653 6902 leader_election.cc:302] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e299f1ab1914a55b731eca71c676695, a7aeb2dd48ef46db8cca98c550ac889e; no voters:
I20260923 23:05:20.963832 7044 raft_consensus.cc:2781] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:20.963883 7044 raft_consensus.cc:482] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:20.963905 7044 raft_consensus.cc:3037] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:20.964843 7044 raft_consensus.cc:504] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:20.964993 7044 leader_election.cc:288] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [CANDIDATE]: Term 1 election: Requested vote from peers 74a58fea6c6f491ba22cd39197a36981 (127.5.74.1:35523), a7aeb2dd48ef46db8cca98c550ac889e (127.5.74.2:35987)
I20260923 23:05:20.965188 6688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96be3cecfc64a49a3c6f039ceed2a2b" candidate_uuid: "0e299f1ab1914a55b731eca71c676695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a58fea6c6f491ba22cd39197a36981"
I20260923 23:05:20.965263 6688 raft_consensus.cc:3037] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:20.965291 6828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96be3cecfc64a49a3c6f039ceed2a2b" candidate_uuid: "0e299f1ab1914a55b731eca71c676695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7aeb2dd48ef46db8cca98c550ac889e"
I20260923 23:05:20.965378 6828 raft_consensus.cc:3037] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:20.966074 6688 raft_consensus.cc:2454] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e299f1ab1914a55b731eca71c676695 in term 1.
I20260923 23:05:20.966094 6828 raft_consensus.cc:2454] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e299f1ab1914a55b731eca71c676695 in term 1.
I20260923 23:05:20.966277 6901 leader_election.cc:302] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e299f1ab1914a55b731eca71c676695, 74a58fea6c6f491ba22cd39197a36981; no voters:
I20260923 23:05:20.966392 7044 raft_consensus.cc:2781] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:20.966581 7044 raft_consensus.cc:686] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 1 LEADER]: Becoming Leader. State: Replica: 0e299f1ab1914a55b731eca71c676695, State: Running, Role: LEADER
I20260923 23:05:20.966658 7044 consensus_queue.cc:237] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:20.967417 6530 catalog_manager.cc:5495] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e299f1ab1914a55b731eca71c676695 (127.5.74.3). New cstate: current_term: 1 leader_uuid: "0e299f1ab1914a55b731eca71c676695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } health_report { overall_health: HEALTHY } } }
Sep 23 23:05:21 dist-test-slave-j4c5 krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:21.018853 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.001516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.5.74.62:41093)
0923 23:05:21.001705 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:21.001707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:21.001926 (+ 219us) server_negotiation.cc:407] Connection header received
0923 23:05:21.001989 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:21.001989 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:21.002002 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:21.002067 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:21.002662 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.003983 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.005093 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.005779 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.005883 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.015950 (+ 10067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:21.015952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:21.015952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:21.015960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:21.017560 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:21.018054 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:21.018055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:21.018055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:21.018104 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:21.018330 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:21.018330 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:21.018331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:21.018455 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:21.018511 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:21.018712 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 23:05:21.018785 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 23:05:21.019037 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.001462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41093 (local address 127.0.0.1:38478)
0923 23:05:21.001797 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:21.001811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:21.001918 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:21.002104 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:21.002105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:21.002450 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:21.002563 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.002566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.004382 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.004383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.004977 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.004979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.005943 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.005944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.005994 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.014040 (+ 8046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:21.014047 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:21.015797 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:21.017666 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:21.017677 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:21.017679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:21.017972 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:21.018176 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:21.018177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:21.018178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:21.018253 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:21.018530 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:21.018538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:21.018597 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 23:05:21.018677 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 23:05:21.018861 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":55,"thread_start_us":253,"threads_started":1}
I20260923 23:05:21.021445 6530 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:38478:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 23:05:21.021672 6530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:21.023190 6808 tablet_service.cc:1467] Processing CreateTablet for tablet f9e9adffa13b45a6a8739187d5622942 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b39a413e4c944bf9ef3f6228b32c41e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:05:21.023190 6948 tablet_service.cc:1467] Processing CreateTablet for tablet f9e9adffa13b45a6a8739187d5622942 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b39a413e4c944bf9ef3f6228b32c41e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:05:21.023198 6668 tablet_service.cc:1467] Processing CreateTablet for tablet f9e9adffa13b45a6a8739187d5622942 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b39a413e4c944bf9ef3f6228b32c41e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:05:21.023355 6808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e9adffa13b45a6a8739187d5622942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:21.023356 6668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e9adffa13b45a6a8739187d5622942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:21.023355 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e9adffa13b45a6a8739187d5622942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:21.024955 7039 tablet_bootstrap.cc:492] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: Bootstrap starting.
I20260923 23:05:21.025002 7038 tablet_bootstrap.cc:492] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: Bootstrap starting.
I20260923 23:05:21.025574 7038 tablet_bootstrap.cc:654] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:21.025667 7039 tablet_bootstrap.cc:654] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:21.025957 7037 tablet_bootstrap.cc:492] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: Bootstrap starting.
I20260923 23:05:21.026386 7039 tablet_bootstrap.cc:492] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: No bootstrap required, opened a new log
I20260923 23:05:21.026410 7037 tablet_bootstrap.cc:654] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:21.026437 7039 ts_tablet_manager.cc:1397] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:21.026614 7039 raft_consensus.cc:348] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:21.026707 7039 raft_consensus.cc:374] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:21.026729 7039 raft_consensus.cc:729] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7aeb2dd48ef46db8cca98c550ac889e, State: Initialized, Role: FOLLOWER
I20260923 23:05:21.026785 7039 consensus_queue.cc:260] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:21.026894 7039 ts_tablet_manager.cc:1428] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:21.026995 7057 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.027100 7057 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:21.027200 7038 tablet_bootstrap.cc:492] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: No bootstrap required, opened a new log
I20260923 23:05:21.027242 7038 ts_tablet_manager.cc:1397] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:05:21.027666 7038 raft_consensus.cc:348] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:21.027750 7038 raft_consensus.cc:374] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:21.027770 7038 raft_consensus.cc:729] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e299f1ab1914a55b731eca71c676695, State: Initialized, Role: FOLLOWER
I20260923 23:05:21.027818 7038 consensus_queue.cc:260] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:21.027913 7038 ts_tablet_manager.cc:1428] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:21.027959 7044 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.028025 7044 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:21.028204 7037 tablet_bootstrap.cc:492] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: No bootstrap required, opened a new log
I20260923 23:05:21.028244 7037 ts_tablet_manager.cc:1397] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:21.028389 7037 raft_consensus.cc:348] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:21.028469 7037 raft_consensus.cc:374] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:21.028496 7037 raft_consensus.cc:729] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a58fea6c6f491ba22cd39197a36981, State: Initialized, Role: FOLLOWER
I20260923 23:05:21.028548 7037 consensus_queue.cc:260] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:21.028643 7037 ts_tablet_manager.cc:1428] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:21.028751 7043 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.028828 7043 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:21.039008 7043 raft_consensus.cc:482] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:21.039108 7043 raft_consensus.cc:504] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:21.039510 7043 leader_election.cc:288] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7aeb2dd48ef46db8cca98c550ac889e (127.5.74.2:35987), 0e299f1ab1914a55b731eca71c676695 (127.5.74.3:34909)
W20260923 23:05:21.040863 7015 tablet.cc:2369] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:21.043152 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.039717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:34909 (local address 127.5.74.1:43429)
0923 23:05:21.040066 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:21.040076 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:21.040194 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:21.040389 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:21.040390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:21.040392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:21.040498 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.040500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.041382 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.041383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.042230 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.042233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.042863 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.042864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.042936 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.042941 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:21.043003 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 23:05:21.043048 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":39,"thread_start_us":81,"threads_started":1}
I20260923 23:05:21.043126 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.039931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:43429 (local address 127.5.74.3:34909)
0923 23:05:21.039976 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:21.039978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:21.040126 (+ 148us) server_negotiation.cc:407] Connection header received
0923 23:05:21.040272 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:21.040274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:21.040284 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:21.040357 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:21.040577 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.041263 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.042340 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.042756 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.042828 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.042838 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:21.043019 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 23:05:21.043056 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:21.043629 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e9adffa13b45a6a8739187d5622942" candidate_uuid: "74a58fea6c6f491ba22cd39197a36981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e299f1ab1914a55b731eca71c676695" is_pre_election: true
I20260923 23:05:21.043746 6968 raft_consensus.cc:2454] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a58fea6c6f491ba22cd39197a36981 in term 0.
I20260923 23:05:21.043933 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.039739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:46681 (local address 127.5.74.2:35987)
0923 23:05:21.039786 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:21.039788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:21.039906 (+ 118us) server_negotiation.cc:407] Connection header received
0923 23:05:21.040061 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:21.040062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:21.040076 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:21.040109 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:21.040510 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.041285 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.042690 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.043254 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.043336 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.043347 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:21.043810 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 23:05:21.043864 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:05:21.043944 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.039625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35987 (local address 127.5.74.1:46681)
0923 23:05:21.039848 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:21.039859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:21.039935 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:21.040229 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:21.040231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:21.040233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:21.040413 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.040416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.041391 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.041392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.042596 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.042599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.043616 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.043618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.043694 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.043698 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:21.043756 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:05:21.043835 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":27,"thread_start_us":94,"threads_started":1}
I20260923 23:05:21.043964 6620 leader_election.cc:302] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e299f1ab1914a55b731eca71c676695, 74a58fea6c6f491ba22cd39197a36981; no voters:
I20260923 23:05:21.044095 7043 raft_consensus.cc:2781] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:21.044127 7043 raft_consensus.cc:482] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:21.044193 7043 raft_consensus.cc:3037] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:21.044260 6828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e9adffa13b45a6a8739187d5622942" candidate_uuid: "74a58fea6c6f491ba22cd39197a36981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" is_pre_election: true
I20260923 23:05:21.044358 6828 raft_consensus.cc:2454] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a58fea6c6f491ba22cd39197a36981 in term 0.
I20260923 23:05:21.045080 7043 raft_consensus.cc:504] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:21.045346 7043 leader_election.cc:288] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [CANDIDATE]: Term 1 election: Requested vote from peers a7aeb2dd48ef46db8cca98c550ac889e (127.5.74.2:35987), 0e299f1ab1914a55b731eca71c676695 (127.5.74.3:34909)
I20260923 23:05:21.045496 6828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e9adffa13b45a6a8739187d5622942" candidate_uuid: "74a58fea6c6f491ba22cd39197a36981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7aeb2dd48ef46db8cca98c550ac889e"
I20260923 23:05:21.045565 6828 raft_consensus.cc:3037] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:21.045715 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e9adffa13b45a6a8739187d5622942" candidate_uuid: "74a58fea6c6f491ba22cd39197a36981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e299f1ab1914a55b731eca71c676695"
I20260923 23:05:21.045781 6968 raft_consensus.cc:3037] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:21.046386 6828 raft_consensus.cc:2454] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a58fea6c6f491ba22cd39197a36981 in term 1.
I20260923 23:05:21.046599 6622 leader_election.cc:302] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a58fea6c6f491ba22cd39197a36981, a7aeb2dd48ef46db8cca98c550ac889e; no voters:
I20260923 23:05:21.046653 6968 raft_consensus.cc:2454] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a58fea6c6f491ba22cd39197a36981 in term 1.
I20260923 23:05:21.046710 7043 raft_consensus.cc:2781] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:21.046850 7043 raft_consensus.cc:686] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [term 1 LEADER]: Becoming Leader. State: Replica: 74a58fea6c6f491ba22cd39197a36981, State: Running, Role: LEADER
I20260923 23:05:21.046936 7043 consensus_queue.cc:237] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } }
I20260923 23:05:21.047662 7080 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: TxnStatusTablet state changed. Reason: New leader 74a58fea6c6f491ba22cd39197a36981. Latest consensus state: current_term: 1 leader_uuid: "74a58fea6c6f491ba22cd39197a36981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.047861 7080 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:05:21.047914 6530 catalog_manager.cc:5495] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a58fea6c6f491ba22cd39197a36981 (127.5.74.1). New cstate: current_term: 1 leader_uuid: "74a58fea6c6f491ba22cd39197a36981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:21.048074 7082 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 23:05:21.048274 7082 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 23:05:21.055637 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.051995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35523 (local address 127.0.0.1:58216)
0923 23:05:21.052077 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:21.052090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:21.052134 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:21.052512 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:21.052513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:21.052515 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:21.052565 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.052568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.053506 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.053507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.054446 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.054449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.054961 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.054962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.055034 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.055056 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:21.055433 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:21.055442 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 23:05:21.055488 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:05:21.055545 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 23:05:21.055951 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.052075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.5.74.1:35523)
0923 23:05:21.052221 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:21.052223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:21.052232 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:21.052383 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:21.052384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:21.052403 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:21.052462 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:21.052639 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.053402 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.054548 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.054855 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.054918 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.055166 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:21.055308 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:21.055395 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:21.055858 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 23:05:21.055881 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260923 23:05:21.057545 6828 raft_consensus.cc:1261] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e [term 1 FOLLOWER]: Refusing update from remote peer 74a58fea6c6f491ba22cd39197a36981: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:21.057569 6968 raft_consensus.cc:1261] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695 [term 1 FOLLOWER]: Refusing update from remote peer 74a58fea6c6f491ba22cd39197a36981: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:21.057917 7043 consensus_queue.cc:1035] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:21.058089 7043 consensus_queue.cc:1035] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:21.058791 7044 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: TxnStatusTablet state changed. Reason: New leader 74a58fea6c6f491ba22cd39197a36981. Latest consensus state: current_term: 1 leader_uuid: "74a58fea6c6f491ba22cd39197a36981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.058892 7044 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:21.058981 7057 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: TxnStatusTablet state changed. Reason: New leader 74a58fea6c6f491ba22cd39197a36981. Latest consensus state: current_term: 1 leader_uuid: "74a58fea6c6f491ba22cd39197a36981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.059094 7057 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:21.061455 7043 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74a58fea6c6f491ba22cd39197a36981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.061550 7043 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:05:21.061591 7043 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74a58fea6c6f491ba22cd39197a36981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.061628 7043 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P 74a58fea6c6f491ba22cd39197a36981: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:05:21.061810 7044 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74a58fea6c6f491ba22cd39197a36981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.061898 7044 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P 0e299f1ab1914a55b731eca71c676695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:21.062795 7057 tablet_replica.cc:439] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74a58fea6c6f491ba22cd39197a36981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 } } peers { permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 } } peers { permanent_uuid: "0e299f1ab1914a55b731eca71c676695" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 34909 } } }
I20260923 23:05:21.062919 7057 tablet_replica.cc:442] T f9e9adffa13b45a6a8739187d5622942 P a7aeb2dd48ef46db8cca98c550ac889e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:21.067287 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.063918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.5.74.3:34909)
0923 23:05:21.063956 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:21.063958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:21.063985 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:05:21.064087 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:21.064087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:21.064108 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:21.064220 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:21.064379 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.065150 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.066315 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.066638 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.066714 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.066897 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:21.067028 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:21.067105 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:21.067160 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 23:05:21.067181 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:05:21.073617 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.070419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:46319 (local address 127.5.74.1:35523)
0923 23:05:21.070462 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:21.070464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:21.070473 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:21.070521 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:21.070522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:21.070533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:21.070586 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:21.070748 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.071483 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.072694 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.073044 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.073130 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.073333 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:21.073369 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:21.073426 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:21.073500 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 23:05:21.073521 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:21.073635 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.070339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35523 (local address 127.5.74.3:46319)
0923 23:05:21.070412 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:21.070423 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:21.070461 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:21.070605 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:21.070606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:21.070607 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:21.070672 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.070673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.071562 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.071563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.072597 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.072600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.073131 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.073132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.073211 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.073239 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:21.073440 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:21.073443 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:21.073487 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 23:05:21.073549 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 23:05:21.076797 6688 raft_consensus.cc:1261] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Refusing update from remote peer 0e299f1ab1914a55b731eca71c676695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:21.076787 6828 raft_consensus.cc:1261] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [term 1 FOLLOWER]: Refusing update from remote peer 0e299f1ab1914a55b731eca71c676695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:21.077050 7067 consensus_queue.cc:1035] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7aeb2dd48ef46db8cca98c550ac889e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:21.077174 7044 consensus_queue.cc:1035] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a58fea6c6f491ba22cd39197a36981" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:21.089694 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.086483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:34909 (local address 127.5.74.1:58513)
0923 23:05:21.086771 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:21.086783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:21.086849 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:21.087001 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:21.087003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:21.087005 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:21.087164 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.087167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.087896 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.087897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.088675 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:21.088677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.089152 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.089154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:21.089235 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.089259 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:21.089473 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:21.089475 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:05:21.089554 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 23:05:21.089604 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":121,"threads_started":1}
I20260923 23:05:21.089686 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.086681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:58513 (local address 127.5.74.3:34909)
0923 23:05:21.086734 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:21.086736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:21.086834 (+ 98us) server_negotiation.cc:407] Connection header received
0923 23:05:21.086909 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:21.086910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:21.086923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:21.086996 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:21.087272 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.087799 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.088782 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.089076 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.089135 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.089338 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:21.089376 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:21.089427 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:21.089568 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 23:05:21.089598 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:05:21.099557 6530 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:21.099687 6530 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:21.100366 6530 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65a7336ab83a4e8bb4f8f9762759ca27: Sending DeleteTablet for 3 replicas of tablet f96be3cecfc64a49a3c6f039ceed2a2b
I20260923 23:05:21.100646 6948 tablet_service.cc:1514] Processing DeleteTablet for tablet f96be3cecfc64a49a3c6f039ceed2a2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:21 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:57470
I20260923 23:05:21.100740 7060 tablet_replica.cc:330] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: stopping tablet replica
I20260923 23:05:21.100785 7060 raft_consensus.cc:2229] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:21.100868 7060 raft_consensus.cc:2258] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:21.101138 6668 tablet_service.cc:1514] Processing DeleteTablet for tablet f96be3cecfc64a49a3c6f039ceed2a2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:21 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:58210
I20260923 23:05:21.101258 7061 tablet_replica.cc:330] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: stopping tablet replica
I20260923 23:05:21.101310 7061 raft_consensus.cc:2229] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:21.101359 7061 raft_consensus.cc:2258] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:21.101471 7060 ts_tablet_manager.cc:1905] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:21.101686 7061 ts_tablet_manager.cc:1905] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:21.103513 7061 ts_tablet_manager.cc:1918] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:21.103541 7060 ts_tablet_manager.cc:1918] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:21.103575 7061 log.cc:1196] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/f96be3cecfc64a49a3c6f039ceed2a2b
I20260923 23:05:21.103600 7060 log.cc:1196] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/f96be3cecfc64a49a3c6f039ceed2a2b
I20260923 23:05:21.103818 7061 ts_tablet_manager.cc:1939] T f96be3cecfc64a49a3c6f039ceed2a2b P 74a58fea6c6f491ba22cd39197a36981: Deleting consensus metadata
I20260923 23:05:21.103835 7060 ts_tablet_manager.cc:1939] T f96be3cecfc64a49a3c6f039ceed2a2b P 0e299f1ab1914a55b731eca71c676695: Deleting consensus metadata
I20260923 23:05:21.104238 6516 catalog_manager.cc:4841] TS 0e299f1ab1914a55b731eca71c676695 (127.5.74.3:34909): tablet f96be3cecfc64a49a3c6f039ceed2a2b (table test-table [id=7abc73b8b216400f8b3340b6c0067c78]) successfully deleted
I20260923 23:05:21.104339 6808 tablet_service.cc:1514] Processing DeleteTablet for tablet f96be3cecfc64a49a3c6f039ceed2a2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:21 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:43540
I20260923 23:05:21.104442 7062 tablet_replica.cc:330] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: stopping tablet replica
I20260923 23:05:21.104511 7062 raft_consensus.cc:2229] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:21.104557 7062 raft_consensus.cc:2258] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:21.104758 6517 catalog_manager.cc:4841] TS 74a58fea6c6f491ba22cd39197a36981 (127.5.74.1:35523): tablet f96be3cecfc64a49a3c6f039ceed2a2b (table test-table [id=7abc73b8b216400f8b3340b6c0067c78]) successfully deleted
I20260923 23:05:21.105129 7062 ts_tablet_manager.cc:1905] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:21.107029 7062 ts_tablet_manager.cc:1918] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:21.107092 7062 log.cc:1196] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/f96be3cecfc64a49a3c6f039ceed2a2b
I20260923 23:05:21.107317 7062 ts_tablet_manager.cc:1939] T f96be3cecfc64a49a3c6f039ceed2a2b P a7aeb2dd48ef46db8cca98c550ac889e: Deleting consensus metadata
I20260923 23:05:21.107702 6518 catalog_manager.cc:4841] TS a7aeb2dd48ef46db8cca98c550ac889e (127.5.74.2:35987): tablet f96be3cecfc64a49a3c6f039ceed2a2b (table test-table [id=7abc73b8b216400f8b3340b6c0067c78]) successfully deleted
Sep 23 23:05:21 dist-test-slave-j4c5 krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:21.119256 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.104209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.5.74.1:35523)
0923 23:05:21.104258 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:21.104261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:21.108393 (+ 4132us) server_negotiation.cc:407] Connection header received
0923 23:05:21.108503 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:21.108504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:21.108518 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:21.108584 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:21.109315 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.110202 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.111310 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.111637 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.111731 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.115648 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:21.115662 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:21.115662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:21.115674 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:21.117862 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:21.118302 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:21.118306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:21.118306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:21.118376 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:21.118597 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:21.118598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:21.118598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:21.118823 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:21.118966 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:21.119057 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 23:05:21.119139 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 23:05:21.119506 6728 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58226
I20260923 23:05:21.130877 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:21.121027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.5.74.62:41093)
0923 23:05:21.121108 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:21.121111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:21.121439 (+ 328us) server_negotiation.cc:407] Connection header received
0923 23:05:21.121564 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:21.121566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:21.121578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:21.121667 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:21.122492 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.123386 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.124672 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:21.125037 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:21.125085 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:21.127136 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:21.127139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:21.127139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:21.127148 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:21.129353 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:21.129885 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:21.129887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:21.129887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:21.129954 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:21.130195 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:21.130196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:21.130196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:21.130344 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:21.130424 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:21.130672 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 23:05:21.130760 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260923 23:05:21.132102 6530 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38492
I20260923 23:05:21.133173 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 6603
I20260923 23:05:21.140378 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 6743
I20260923 23:05:21.147944 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 6883
I20260923 23:05:21.153644 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 6499
2026-09-23T23:05:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2280 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:21 dist-test-slave-j4c5 krb5kdc[7103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:21 dist-test-slave-j4c5 krb5kdc[7103](info): set up 2 sockets
Sep 23 23:05:21 dist-test-slave-j4c5 krb5kdc[7103](info): commencing operation
krb5kdc: starting...
W20260923 23:05:22.248440 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:22 dist-test-slave-j4c5 krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:22.389649 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:36053
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:36053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:22.479455 7116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:22.479727 7116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:22.479769 7116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:22.481114 7116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:22.481191 7116 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 23:05:22.481215 7116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:22.481247 7116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:22.481276 7116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:22.481300 7116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:22.483142 7116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:36053
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:36053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:22.483489 7116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:22.483744 7116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:22.489559 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:22.489569 7124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:22.489569 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:22.490093 7116 server_base.cc:1034] running on GCE node
I20260923 23:05:22.490351 7116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:22.490679 7116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:22.491961 7116 hybrid_clock.cc:648] HybridClock initialized: now 1790204722491912 us; error 40 us; skew 500 ppm
Sep 23 23:05:22 dist-test-slave-j4c5 krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:22.494796 7116 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:22.495402 7116 webserver.cc:460] Webserver started at http://127.5.74.62:41359/ using document root <none> and password file <none>
I20260923 23:05:22.495645 7116 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:22.495714 7116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:22.495854 7116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:22.497087 7116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "285f7200162c44f889d11cdca9f1ca48"
format_stamp: "Formatted at 2026-09-23 23:05:22 on dist-test-slave-j4c5"
server_key: "e8d7b825c603d9f438ce57854dcf6de9"
server_key_iv: "e6eebca6e8a8108d49f7730e23bab957"
server_key_version: "encryptionkey@0"
I20260923 23:05:22.497470 7116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "285f7200162c44f889d11cdca9f1ca48"
format_stamp: "Formatted at 2026-09-23 23:05:22 on dist-test-slave-j4c5"
server_key: "e8d7b825c603d9f438ce57854dcf6de9"
server_key_iv: "e6eebca6e8a8108d49f7730e23bab957"
server_key_version: "encryptionkey@0"
I20260923 23:05:22.498986 7116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:05:22.499835 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:22.500108 7116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:22.500257 7116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "285f7200162c44f889d11cdca9f1ca48"
format_stamp: "Formatted at 2026-09-23 23:05:22 on dist-test-slave-j4c5"
server_key: "e8d7b825c603d9f438ce57854dcf6de9"
server_key_iv: "e6eebca6e8a8108d49f7730e23bab957"
server_key_version: "encryptionkey@0"
I20260923 23:05:22.500340 7116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:22.517189 7116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:22.519459 7116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:22.519645 7116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:22.524581 7116 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:36053
I20260923 23:05:22.524809 7193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:36053 every 8 connection(s)
I20260923 23:05:22.525049 7116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:22.525949 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:22.526013 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 7116
I20260923 23:05:22.526095 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:22.526286 5416 external_mini_cluster.cc:1428] Setting key c2fd920fec29f3de12e47daf67e547c3
I20260923 23:05:22.529902 7194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48: Bootstrap starting.
I20260923 23:05:22.530968 7194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:22.531383 7194 log.cc:824] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:22.532542 7194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48: No bootstrap required, opened a new log
Sep 23 23:05:22 dist-test-slave-j4c5 krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:22.535482 7194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285f7200162c44f889d11cdca9f1ca48" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36053 } }
I20260923 23:05:22.535656 7194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:22.535679 7194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285f7200162c44f889d11cdca9f1ca48, State: Initialized, Role: FOLLOWER
I20260923 23:05:22.535861 7194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285f7200162c44f889d11cdca9f1ca48" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36053 } }
I20260923 23:05:22.535959 7194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:22.536019 7194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:22.536094 7194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:22.536752 7194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285f7200162c44f889d11cdca9f1ca48" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36053 } }
I20260923 23:05:22.536895 7194 leader_election.cc:302] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 285f7200162c44f889d11cdca9f1ca48; no voters:
I20260923 23:05:22.537091 7194 leader_election.cc:288] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:22.537142 7199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:22.537303 7199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [term 1 LEADER]: Becoming Leader. State: Replica: 285f7200162c44f889d11cdca9f1ca48, State: Running, Role: LEADER
I20260923 23:05:22.537397 7194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:22.537410 7199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285f7200162c44f889d11cdca9f1ca48" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36053 } }
I20260923 23:05:22.537782 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "285f7200162c44f889d11cdca9f1ca48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285f7200162c44f889d11cdca9f1ca48" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36053 } } }
I20260923 23:05:22.537864 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:22.538021 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 285f7200162c44f889d11cdca9f1ca48. Latest consensus state: current_term: 1 leader_uuid: "285f7200162c44f889d11cdca9f1ca48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285f7200162c44f889d11cdca9f1ca48" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36053 } } }
I20260923 23:05:22.538089 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:22.538221 7204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:22.538758 7204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:22.540655 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:22.526868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.5.74.62:36053)
0923 23:05:22.527044 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:22.527060 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:22.527092 (+ 32us) server_negotiation.cc:407] Connection header received
0923 23:05:22.527864 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:22.527871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:22.527922 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:22.528320 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:22.529166 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.530425 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.532015 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.532533 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.532653 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:22.535845 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:22.535851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:22.535851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:22.535865 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:22.538423 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.539073 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.539076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.539077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.539200 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.539521 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.539523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.539523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.539843 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:22.539976 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:22.540171 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 23:05:22.540329 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260923 23:05:22.540719 7204 catalog_manager.cc:1347] Generated new cluster ID: 9a9264ab915a4890aef0555bbf103031
I20260923 23:05:22.540769 7204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:22.558259 7204 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:22.558846 7204 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:22.564337 7204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48: Generated new TSK 0
I20260923 23:05:22.564594 7204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:22.605940 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:36053
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:22.695329 7220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:22.695605 7220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:22.695647 7220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:22.697037 7220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:22.697137 7220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:22.697173 7220 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 23:05:22.697193 7220 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:05:22.697237 7220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:22.699440 7220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.74.62:36053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:22.699788 7220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:22.700217 7220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:22.706200 7228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:22.706202 7226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:22.706341 7220 server_base.cc:1034] running on GCE node
W20260923 23:05:22.706200 7225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:22.706755 7220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:22.707049 7220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:22.708268 7220 hybrid_clock.cc:648] HybridClock initialized: now 1790204722708190 us; error 99 us; skew 500 ppm
Sep 23 23:05:22 dist-test-slave-j4c5 krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:22.710981 7220 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:22.711459 7220 webserver.cc:460] Webserver started at http://127.5.74.1:39903/ using document root <none> and password file <none>
I20260923 23:05:22.711826 7220 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:22.711907 7220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:22.712071 7220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:22.713217 7220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "8770c69dfafd4992bf0fb87067f162af"
format_stamp: "Formatted at 2026-09-23 23:05:22 on dist-test-slave-j4c5"
server_key: "bf06ce6910247ac63840400522be8547"
server_key_iv: "c7a48ab201d5be6ba669b54fb0d2ed9a"
server_key_version: "encryptionkey@0"
I20260923 23:05:22.713831 7220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "8770c69dfafd4992bf0fb87067f162af"
format_stamp: "Formatted at 2026-09-23 23:05:22 on dist-test-slave-j4c5"
server_key: "bf06ce6910247ac63840400522be8547"
server_key_iv: "c7a48ab201d5be6ba669b54fb0d2ed9a"
server_key_version: "encryptionkey@0"
I20260923 23:05:22.715344 7220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:22.716337 7235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:22.716692 7220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:22.716838 7220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "8770c69dfafd4992bf0fb87067f162af"
format_stamp: "Formatted at 2026-09-23 23:05:22 on dist-test-slave-j4c5"
server_key: "bf06ce6910247ac63840400522be8547"
server_key_iv: "c7a48ab201d5be6ba669b54fb0d2ed9a"
server_key_version: "encryptionkey@0"
I20260923 23:05:22.716957 7220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:22.734969 7220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:22.737666 7220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:22.737852 7220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:22.738489 7220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:22.738552 7220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:22.738597 7220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:22.738646 7220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:22.746690 7220 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:44717
I20260923 23:05:22.746780 7350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:44717 every 8 connection(s)
Sep 23 23:05:22 dist-test-slave-j4c5 krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:22.747114 7220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:22.752250 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:22.739001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:58393 (local address 127.5.74.62:36053)
0923 23:05:22.739079 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:22.739082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:22.739437 (+ 355us) server_negotiation.cc:407] Connection header received
0923 23:05:22.740244 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:22.740246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:22.740257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:22.740379 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:22.741485 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.742338 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.744261 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.744505 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.744595 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:22.747885 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:22.747887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:22.747888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:22.747896 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:22.749697 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.750928 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.750931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.750932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.751011 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.751433 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.751434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.751435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.751601 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:22.751690 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:22.752003 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 23:05:22.752119 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 23:05:22.752424 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:22.739057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36053 (local address 127.5.74.1:58393)
0923 23:05:22.739266 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:22.739339 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:22.740067 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:22.740473 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:22.740478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:22.740910 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:22.741247 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:22.741254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.742569 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.742570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:22.744102 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:22.744105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.744595 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.744596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:22.744759 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:22.745525 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:22.745538 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:22.747735 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:22.750035 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:22.750046 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:22.750049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:22.750603 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:22.751097 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:22.751098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:22.751099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:22.751288 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:22.751680 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:22.751687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:22.751855 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 23:05:22.751977 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 23:05:22.752158 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":125,"threads_started":1}
I20260923 23:05:22.752516 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 7220
I20260923 23:05:22.752632 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:22.752825 5416 external_mini_cluster.cc:1428] Setting key 952ce4433a0e50ec126a6a2f0894af6d
I20260923 23:05:22.753002 7351 heartbeater.cc:347] Connected to a master server at 127.5.74.62:36053
I20260923 23:05:22.753104 7351 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:22.753302 7351 heartbeater.cc:511] Master 127.5.74.62:36053 requested a full tablet report, sending...
I20260923 23:05:22.753854 7148 ts_manager.cc:194] Registered new tserver with Master: 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717)
I20260923 23:05:22.754668 7148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:58393
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:05:22.761005 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:22.754973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:37123 (local address 127.5.74.62:36053)
0923 23:05:22.755036 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:22.755038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:22.755047 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:22.755065 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:22.755066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:22.755076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:22.755157 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:22.755490 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.755968 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.756763 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.756974 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.757062 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:22.758161 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:22.758163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:22.758163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:22.758170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:22.759746 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.760145 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.760146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.760147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.760235 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.760449 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.760450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.760451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.760567 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:22.760638 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:22.760823 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 23:05:22.760907 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 23:05:22.761019 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:22.754081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36053 (local address 127.5.74.1:37123)
0923 23:05:22.754260 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:22.754272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:22.754326 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:22.755162 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:22.755163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:22.755360 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:22.755411 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:22.755413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.756080 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.756081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:22.756674 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:22.756676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.757064 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.757065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:22.757141 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:22.757677 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:22.757681 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:22.758030 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:22.759867 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:22.759869 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:22.759869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:22.760055 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:22.760306 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:22.760307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:22.760307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:22.760369 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:22.760639 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:22.760640 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:22.760686 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 23:05:22.760799 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 23:05:22.760909 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:22.785285 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:36053
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:22.870704 7360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:22.870930 7360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:22.870958 7360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:22.872241 7360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:22.872291 7360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:22.872314 7360 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 23:05:22.872326 7360 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:05:22.872345 7360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:22.873895 7360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.74.62:36053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:22.874162 7360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:22.874426 7360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:22.879799 7366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:22.879818 7368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:22.879911 7365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:22.880441 7360 server_base.cc:1034] running on GCE node
I20260923 23:05:22.880662 7360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:22.881000 7360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:22.882189 7360 hybrid_clock.cc:648] HybridClock initialized: now 1790204722882169 us; error 42 us; skew 500 ppm
Sep 23 23:05:22 dist-test-slave-j4c5 krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:22.885090 7360 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:22.885656 7360 webserver.cc:460] Webserver started at http://127.5.74.2:33165/ using document root <none> and password file <none>
I20260923 23:05:22.885931 7360 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:22.886013 7360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:22.886171 7360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:22.887269 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "1f0626a976514613829699d7d07d0adf"
format_stamp: "Formatted at 2026-09-23 23:05:22 on dist-test-slave-j4c5"
server_key: "1b5129470a1dfb067931a8306e461bcb"
server_key_iv: "5ed444b1cccc6d2c362ab522644a9d1f"
server_key_version: "encryptionkey@0"
I20260923 23:05:22.887655 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "1f0626a976514613829699d7d07d0adf"
format_stamp: "Formatted at 2026-09-23 23:05:22 on dist-test-slave-j4c5"
server_key: "1b5129470a1dfb067931a8306e461bcb"
server_key_iv: "5ed444b1cccc6d2c362ab522644a9d1f"
server_key_version: "encryptionkey@0"
I20260923 23:05:22.889153 7360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:05:22.889992 7375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:22.890218 7360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:05:22.890317 7360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "1f0626a976514613829699d7d07d0adf"
format_stamp: "Formatted at 2026-09-23 23:05:22 on dist-test-slave-j4c5"
server_key: "1b5129470a1dfb067931a8306e461bcb"
server_key_iv: "5ed444b1cccc6d2c362ab522644a9d1f"
server_key_version: "encryptionkey@0"
I20260923 23:05:22.890398 7360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:22.913410 7360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:22.916071 7360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:22.916293 7360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:22.916882 7360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:22.916960 7360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:22.917021 7360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:22.917052 7360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:22.924897 7360 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:38761
I20260923 23:05:22.924984 7490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:38761 every 8 connection(s)
I20260923 23:05:22.925313 7360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:05:22 dist-test-slave-j4c5 krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:22.930090 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:22.917473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:38247 (local address 127.5.74.62:36053)
0923 23:05:22.917537 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:22.917539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:22.917792 (+ 253us) server_negotiation.cc:407] Connection header received
0923 23:05:22.918329 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:22.918330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:22.918345 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:22.918396 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:22.919263 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.920056 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.922129 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.922345 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.922440 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:22.927030 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:22.927032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:22.927032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:22.927041 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:22.928640 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.929095 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.929097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.929097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.929159 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.929423 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.929424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.929425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.929549 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:22.929626 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:22.929915 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 23:05:22.929997 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 23:05:22.930305 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:22.917472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36053 (local address 127.5.74.2:38247)
0923 23:05:22.917633 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:22.917671 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:22.918213 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:22.918471 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:22.918475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:22.918845 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:22.919170 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:22.919178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.920201 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.920202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:22.922018 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:22.922022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.922463 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.922464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:22.922602 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:22.923214 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:22.923223 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:22.926904 (+ 3681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:22.928744 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:22.928751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:22.928754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:22.929003 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:22.929227 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:22.929227 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:22.929228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:22.929351 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:22.929616 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:22.929622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:22.929774 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 23:05:22.929882 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 23:05:22.930054 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260923 23:05:22.930867 7491 heartbeater.cc:347] Connected to a master server at 127.5.74.62:36053
I20260923 23:05:22.930958 7491 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:22.931173 7491 heartbeater.cc:511] Master 127.5.74.62:36053 requested a full tablet report, sending...
I20260923 23:05:22.931236 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 7360
I20260923 23:05:22.931360 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:22.931547 5416 external_mini_cluster.cc:1428] Setting key 317b036d2037d12c531b821a446c31e1
I20260923 23:05:22.931955 7148 ts_manager.cc:194] Registered new tserver with Master: 1f0626a976514613829699d7d07d0adf (127.5.74.2:38761)
I20260923 23:05:22.932723 7148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:38247
I20260923 23:05:22.939231 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:22.932118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:46123 (local address 127.5.74.62:36053)
0923 23:05:22.932196 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:22.932198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:22.932206 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:22.932227 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:22.932228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:22.932237 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:22.932452 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:22.932942 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.933590 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.934467 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.934772 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.934857 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:22.935995 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:22.935997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:22.935997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:22.936003 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:22.937635 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.938161 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.938162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.938162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.938245 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:22.938519 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:22.938520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:22.938520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:22.938735 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:22.938876 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:22.939073 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 23:05:22.939165 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:05:22.939565 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:22.931902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36053 (local address 127.5.74.2:46123)
0923 23:05:22.932059 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:22.932070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:22.932136 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:22.932508 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:22.932509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:22.932754 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:22.932833 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:22.932836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.933698 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.933699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:22.934382 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:22.934384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:22.934881 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:22.934882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:22.934933 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:22.935441 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:22.935446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:22.935883 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:22.937748 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:22.937749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:22.937750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:22.938002 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:22.938328 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:22.938329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:22.938329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:22.938415 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:22.938871 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:22.938873 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:22.938928 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 23:05:22.939018 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 23:05:22.939137 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:22.965333 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:36053
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:23.050119 7500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:23.050390 7500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:23.050432 7500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:23.051663 7500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:23.051738 7500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:23.051775 7500 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 23:05:23.051802 7500 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:05:23.051834 7500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:23.053545 7500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.74.62:36053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:23.053825 7500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:23.054096 7500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:23.059777 7505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:23.059755 7508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:23.059752 7506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:23.059832 7500 server_base.cc:1034] running on GCE node
I20260923 23:05:23.060364 7500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:23.060647 7500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:23.061866 7500 hybrid_clock.cc:648] HybridClock initialized: now 1790204723061837 us; error 45 us; skew 500 ppm
Sep 23 23:05:23 dist-test-slave-j4c5 krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204723, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:23.064816 7500 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:23.065407 7500 webserver.cc:460] Webserver started at http://127.5.74.3:34491/ using document root <none> and password file <none>
I20260923 23:05:23.065654 7500 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:23.065726 7500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:23.065876 7500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:23.066964 7500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "d63040bc26b54b60baf84b93f0aab814"
format_stamp: "Formatted at 2026-09-23 23:05:23 on dist-test-slave-j4c5"
server_key: "80bd986dfc69ee9dfbdbfac67cd00c39"
server_key_iv: "dfe67f4a5710272d86f898ddafd18cf7"
server_key_version: "encryptionkey@0"
I20260923 23:05:23.067341 7500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "d63040bc26b54b60baf84b93f0aab814"
format_stamp: "Formatted at 2026-09-23 23:05:23 on dist-test-slave-j4c5"
server_key: "80bd986dfc69ee9dfbdbfac67cd00c39"
server_key_iv: "dfe67f4a5710272d86f898ddafd18cf7"
server_key_version: "encryptionkey@0"
I20260923 23:05:23.068868 7500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:23.069702 7515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:23.069983 7500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:23.070089 7500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "d63040bc26b54b60baf84b93f0aab814"
format_stamp: "Formatted at 2026-09-23 23:05:23 on dist-test-slave-j4c5"
server_key: "80bd986dfc69ee9dfbdbfac67cd00c39"
server_key_iv: "dfe67f4a5710272d86f898ddafd18cf7"
server_key_version: "encryptionkey@0"
I20260923 23:05:23.070181 7500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:23.084438 7500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:23.086920 7500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:23.087121 7500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:23.087966 7500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:23.088043 7500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:23.088099 7500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:23.088125 7500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:05:23 dist-test-slave-j4c5 krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204723, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:23.096904 7500 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:33643
I20260923 23:05:23.096900 7630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:33643 every 8 connection(s)
I20260923 23:05:23.097373 7500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:05:23.099989 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.088674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:57329 (local address 127.5.74.62:36053)
0923 23:05:23.088894 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.088897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.089057 (+ 160us) server_negotiation.cc:407] Connection header received
0923 23:05:23.089626 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.089628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.089641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.089699 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:23.090456 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.091321 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.092839 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.093045 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.093135 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.096945 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:23.096947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:23.096947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:23.096956 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:23.098617 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:23.099055 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:23.099056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:23.099057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:23.099112 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:23.099358 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:23.099359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:23.099359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:23.099515 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:23.099570 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.099840 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.099919 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260923 23:05:23.100136 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.088631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36053 (local address 127.5.74.3:57329)
0923 23:05:23.088935 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.088972 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.089515 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.089778 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.089782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.090036 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:23.090328 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.090333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.091462 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.091463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.092729 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.092732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.093146 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.093147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.093284 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.093927 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:23.093942 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:23.096789 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:23.098712 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:23.098716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:23.098719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:23.098978 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:23.099201 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:23.099201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:23.099202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:23.099301 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:23.099574 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:23.099578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:23.099697 (+ 119us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.099783 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.099917 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260923 23:05:23.100636 7631 heartbeater.cc:347] Connected to a master server at 127.5.74.62:36053
I20260923 23:05:23.100725 7631 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:23.100946 7631 heartbeater.cc:511] Master 127.5.74.62:36053 requested a full tablet report, sending...
I20260923 23:05:23.101292 7147 ts_manager.cc:194] Registered new tserver with Master: d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643)
I20260923 23:05:23.101688 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 7500
I20260923 23:05:23.101770 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:23.101822 7147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:57329
I20260923 23:05:23.101969 5416 external_mini_cluster.cc:1428] Setting key aa97b247d643c4b7d1f1d0ec56fa2613
I20260923 23:05:23.103348 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:05:23.108916 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.101675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:51909 (local address 127.5.74.62:36053)
0923 23:05:23.101714 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.101716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.102237 (+ 521us) server_negotiation.cc:407] Connection header received
0923 23:05:23.102284 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.102285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.102296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.102351 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:23.102756 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.103472 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.104505 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.104701 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.104758 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.105728 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:23.105730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:23.105730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:23.105738 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:23.107595 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:23.107950 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:23.107952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:23.107953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:23.108027 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:23.108301 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:23.108302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:23.108303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:23.108461 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:23.108552 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.108699 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.108784 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:05:23.108814 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.101643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36053 (local address 127.5.74.3:51909)
0923 23:05:23.102157 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.102169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.102237 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.102354 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.102355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.102549 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:23.102604 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.102606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.103569 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.103570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.104423 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.104426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.104775 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.104776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.104843 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.105283 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:23.105289 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:23.105637 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:23.107696 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:23.107697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:23.107697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:23.107866 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:23.108105 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:23.108106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:23.108106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:23.108229 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:23.108554 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:23.108555 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:23.108595 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.108653 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.108733 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":128,"threads_started":1}
Sep 23 23:05:23 dist-test-slave-j4c5 krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:05:23 dist-test-slave-j4c5 krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:23.130609 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.121651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.5.74.62:36053)
0923 23:05:23.121748 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.121751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.121761 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:23.121800 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.121801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.121832 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.121906 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:23.122772 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.123666 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.124749 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.125042 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.125130 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.127936 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:23.127938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:23.127938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:23.127946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:23.129306 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:23.129674 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:23.129676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:23.129676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:23.129735 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:23.130060 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:23.130061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:23.130061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:23.130187 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:23.130277 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.130463 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.130539 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 23:05:23.132650 7147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:23.134254 7147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:23.141014 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.136638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:33643 (local address 127.0.0.1:56038)
0923 23:05:23.136893 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.136934 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.137012 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.137259 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.137261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.137263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:23.137354 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.137357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.138882 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.138883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.139823 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.139826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.140705 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.140706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.140785 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.140799 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.140869 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.140922 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":62,"threads_started":1}
I20260923 23:05:23.141031 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.136717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.5.74.3:33643)
0923 23:05:23.136882 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.136887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.137103 (+ 216us) server_negotiation.cc:407] Connection header received
0923 23:05:23.137149 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.137152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.137168 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.137258 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:23.137447 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.138778 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.139914 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.140614 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.140691 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.140761 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.140889 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.140929 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260923 23:05:23.141430 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.136716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:38761 (local address 127.0.0.1:45336)
0923 23:05:23.136974 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.136985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.137045 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.137614 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.137615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.137617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:23.137714 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.137716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.138651 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.138652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.140022 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.140025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.141167 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.141168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.141232 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.141236 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.141303 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.141343 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":6,"thread_start_us":94,"threads_started":1}
I20260923 23:05:23.141692 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.136985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.5.74.1:44717)
0923 23:05:23.137168 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.137172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.137183 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:23.137219 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.137221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.137235 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.137308 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:23.137522 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.138759 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.140413 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.141144 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.141214 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.141268 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.141511 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.141577 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260923 23:05:23.141872 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.136717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:44717 (local address 127.0.0.1:57118)
0923 23:05:23.137119 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.137127 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.137160 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.137389 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.137390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.137392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:23.137438 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.137439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.138855 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.138856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.140333 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.140335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.141298 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.141299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.141350 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.141353 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.141397 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.141429 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":34,"thread_start_us":124,"threads_started":1}
I20260923 23:05:23.141884 7565 tablet_service.cc:1467] Processing CreateTablet for tablet 4232bd203eb942c7a20326c86a53ce22 (DEFAULT_TABLE table=test-table [id=d410ec68429942ecb8d7469373f59255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:23.142237 7565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4232bd203eb942c7a20326c86a53ce22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:23.144502 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.136968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.5.74.2:38761)
0923 23:05:23.137414 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.137418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.137427 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:23.137462 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.137464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.137480 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.137551 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:23.137773 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.138566 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.140112 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.140802 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.140876 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.140940 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.144294 (+ 3354us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.144391 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":77,"threads_started":1}
I20260923 23:05:23.144971 7654 tablet_bootstrap.cc:492] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: Bootstrap starting.
I20260923 23:05:23.144905 7285 tablet_service.cc:1467] Processing CreateTablet for tablet 4232bd203eb942c7a20326c86a53ce22 (DEFAULT_TABLE table=test-table [id=d410ec68429942ecb8d7469373f59255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:23.145061 7425 tablet_service.cc:1467] Processing CreateTablet for tablet 4232bd203eb942c7a20326c86a53ce22 (DEFAULT_TABLE table=test-table [id=d410ec68429942ecb8d7469373f59255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:23.145259 7285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4232bd203eb942c7a20326c86a53ce22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:23.145367 7425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4232bd203eb942c7a20326c86a53ce22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:23.145833 7654 tablet_bootstrap.cc:654] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:23.146135 7654 log.cc:824] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:23.146847 7654 tablet_bootstrap.cc:492] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: No bootstrap required, opened a new log
I20260923 23:05:23.146926 7654 ts_tablet_manager.cc:1397] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:23.148520 7654 raft_consensus.cc:348] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } }
I20260923 23:05:23.148720 7654 raft_consensus.cc:374] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:23.148765 7654 raft_consensus.cc:729] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63040bc26b54b60baf84b93f0aab814, State: Initialized, Role: FOLLOWER
I20260923 23:05:23.148897 7654 consensus_queue.cc:260] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } }
I20260923 23:05:23.149183 7654 ts_tablet_manager.cc:1428] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:05:23.149221 7631 heartbeater.cc:503] Master 127.5.74.62:36053 was elected leader, sending a full tablet report...
I20260923 23:05:23.149387 7657 tablet_bootstrap.cc:492] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: Bootstrap starting.
I20260923 23:05:23.149972 7657 tablet_bootstrap.cc:654] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:23.150264 7657 log.cc:824] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:23.150560 7656 tablet_bootstrap.cc:492] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: Bootstrap starting.
I20260923 23:05:23.150974 7657 tablet_bootstrap.cc:492] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: No bootstrap required, opened a new log
I20260923 23:05:23.151031 7657 ts_tablet_manager.cc:1397] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:23.151079 7656 tablet_bootstrap.cc:654] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:23.151365 7656 log.cc:824] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:23.152099 7656 tablet_bootstrap.cc:492] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: No bootstrap required, opened a new log
I20260923 23:05:23.152206 7656 ts_tablet_manager.cc:1397] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:05:23.152976 7657 raft_consensus.cc:348] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } }
I20260923 23:05:23.153110 7657 raft_consensus.cc:374] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:23.153132 7657 raft_consensus.cc:729] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0626a976514613829699d7d07d0adf, State: Initialized, Role: FOLLOWER
I20260923 23:05:23.153257 7657 consensus_queue.cc:260] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } }
I20260923 23:05:23.153584 7657 ts_tablet_manager.cc:1428] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:05:23.153610 7491 heartbeater.cc:503] Master 127.5.74.62:36053 was elected leader, sending a full tablet report...
I20260923 23:05:23.153913 7656 raft_consensus.cc:348] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } }
I20260923 23:05:23.154063 7656 raft_consensus.cc:374] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:23.154107 7656 raft_consensus.cc:729] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8770c69dfafd4992bf0fb87067f162af, State: Initialized, Role: FOLLOWER
I20260923 23:05:23.154225 7656 consensus_queue.cc:260] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } }
I20260923 23:05:23.154549 7351 heartbeater.cc:503] Master 127.5.74.62:36053 was elected leader, sending a full tablet report...
I20260923 23:05:23.154579 7656 ts_tablet_manager.cc:1428] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260923 23:05:23.176043 7492 tablet.cc:2369] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:23.242871 7661 raft_consensus.cc:482] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:23.243005 7661 raft_consensus.cc:504] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } }
I20260923 23:05:23.243589 7661 leader_election.cc:288] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643), 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717)
I20260923 23:05:23.246626 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.243796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:33643 (local address 127.5.74.2:54547)
0923 23:05:23.243839 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.243850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.243910 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.244129 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.244130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.244132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:23.244228 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.244230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.244936 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.244937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.245674 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.245676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.246381 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.246382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.246459 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.246463 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.246512 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.246547 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260923 23:05:23.246778 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.243951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:40441 (local address 127.5.74.1:44717)
0923 23:05:23.244005 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.244007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.244109 (+ 102us) server_negotiation.cc:407] Connection header received
0923 23:05:23.244226 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.244227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.244240 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.244378 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:23.244543 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.245011 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.245915 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.246356 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.246428 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.246438 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.246651 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.246705 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:05:23.246665 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.243942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:54547 (local address 127.5.74.3:33643)
0923 23:05:23.243988 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.243990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.243998 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:23.244027 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.244028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.244039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.244111 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:23.244329 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.244808 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.245770 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.246167 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.246381 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.246413 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.246534 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.246584 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:05:23.246817 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.243832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:44717 (local address 127.5.74.2:40441)
0923 23:05:23.244034 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.244043 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.244124 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.244387 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.244388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.244390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:23.244470 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.244471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.245099 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.245101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.245822 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.245824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.246455 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.246456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.246559 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.246564 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.246632 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.246670 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
I20260923 23:05:23.247053 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4232bd203eb942c7a20326c86a53ce22" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814" is_pre_election: true
I20260923 23:05:23.247143 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4232bd203eb942c7a20326c86a53ce22" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770c69dfafd4992bf0fb87067f162af" is_pre_election: true
I20260923 23:05:23.247339 7305 raft_consensus.cc:2454] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 0.
I20260923 23:05:23.247228 7585 raft_consensus.cc:2454] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 0.
I20260923 23:05:23.247597 7379 leader_election.cc:302] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0626a976514613829699d7d07d0adf, 8770c69dfafd4992bf0fb87067f162af; no voters:
W20260923 23:05:23.248030 7352 tablet.cc:2369] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:23.248294 7661 raft_consensus.cc:2781] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:23.248368 7661 raft_consensus.cc:482] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:23.248400 7661 raft_consensus.cc:3037] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:23.249349 7661 raft_consensus.cc:504] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } }
I20260923 23:05:23.249522 7661 leader_election.cc:288] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 election: Requested vote from peers d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643), 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717)
I20260923 23:05:23.249701 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4232bd203eb942c7a20326c86a53ce22" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814"
I20260923 23:05:23.249686 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4232bd203eb942c7a20326c86a53ce22" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770c69dfafd4992bf0fb87067f162af"
I20260923 23:05:23.249794 7585 raft_consensus.cc:3037] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:23.249806 7305 raft_consensus.cc:3037] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:23.250785 7305 raft_consensus.cc:2454] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 1.
I20260923 23:05:23.250841 7585 raft_consensus.cc:2454] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 1.
I20260923 23:05:23.251080 7378 leader_election.cc:302] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0626a976514613829699d7d07d0adf, d63040bc26b54b60baf84b93f0aab814; no voters:
I20260923 23:05:23.251199 7661 raft_consensus.cc:2781] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:23.251363 7661 raft_consensus.cc:686] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 1 LEADER]: Becoming Leader. State: Replica: 1f0626a976514613829699d7d07d0adf, State: Running, Role: LEADER
I20260923 23:05:23.251463 7661 consensus_queue.cc:237] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } }
I20260923 23:05:23.252449 7147 catalog_manager.cc:5495] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0626a976514613829699d7d07d0adf (127.5.74.2). New cstate: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } health_report { overall_health: UNKNOWN } } }
Sep 23 23:05:23 dist-test-slave-j4c5 krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:23.284514 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.277126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.5.74.62:36053)
0923 23:05:23.277196 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.277198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.277208 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:23.277266 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.277267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.277277 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.277346 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:23.277798 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.278276 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.279059 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.279235 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.279314 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.281706 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:23.281707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:23.281708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:23.281715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:23.283174 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:23.283638 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:23.283640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:23.283640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:23.283698 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:23.283909 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:23.283911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:23.283911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:23.284030 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:23.284108 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.284281 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.284397 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 23:05:23.284591 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.276910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36053 (local address 127.0.0.1:45408)
0923 23:05:23.277097 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.277108 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.277181 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.277352 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.277353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.277668 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:23.277712 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.277714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.278355 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.278356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.278952 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.278955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.279316 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.279317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.279388 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.280002 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:23.280007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:23.281607 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:23.283293 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:23.283303 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:23.283305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:23.283530 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:23.283765 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:23.283767 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:23.283767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:23.283838 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:23.284096 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:23.284104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:23.284178 (+ 74us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.284250 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.284458 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":98,"thread_start_us":108,"threads_started":1}
I20260923 23:05:23.286986 7147 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:45408:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 23:05:23.287181 7147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:23.288887 7425 tablet_service.cc:1467] Processing CreateTablet for tablet a738ffee247e4d1ab1473f1f548259a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efd352de178b42aeaf89d527e9aafd10]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:05:23.288880 7565 tablet_service.cc:1467] Processing CreateTablet for tablet a738ffee247e4d1ab1473f1f548259a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efd352de178b42aeaf89d527e9aafd10]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:05:23.288875 7285 tablet_service.cc:1467] Processing CreateTablet for tablet a738ffee247e4d1ab1473f1f548259a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efd352de178b42aeaf89d527e9aafd10]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:05:23.289068 7285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a738ffee247e4d1ab1473f1f548259a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:23.289068 7565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a738ffee247e4d1ab1473f1f548259a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:23.289070 7425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a738ffee247e4d1ab1473f1f548259a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:23.290582 7654 tablet_bootstrap.cc:492] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: Bootstrap starting.
I20260923 23:05:23.290756 7657 tablet_bootstrap.cc:492] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: Bootstrap starting.
I20260923 23:05:23.291149 7657 tablet_bootstrap.cc:654] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:23.291162 7654 tablet_bootstrap.cc:654] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:23.291486 7656 tablet_bootstrap.cc:492] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: Bootstrap starting.
I20260923 23:05:23.291762 7656 tablet_bootstrap.cc:654] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:23.291810 7654 tablet_bootstrap.cc:492] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: No bootstrap required, opened a new log
I20260923 23:05:23.291857 7654 ts_tablet_manager.cc:1397] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:23.292007 7654 raft_consensus.cc:348] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.292110 7654 raft_consensus.cc:374] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:23.292343 7657 tablet_bootstrap.cc:492] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: No bootstrap required, opened a new log
I20260923 23:05:23.292410 7657 ts_tablet_manager.cc:1397] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:23.292485 7656 tablet_bootstrap.cc:492] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: No bootstrap required, opened a new log
I20260923 23:05:23.292538 7656 ts_tablet_manager.cc:1397] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:23.292574 7657 raft_consensus.cc:348] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.292659 7657 raft_consensus.cc:374] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:23.292682 7657 raft_consensus.cc:729] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0626a976514613829699d7d07d0adf, State: Initialized, Role: FOLLOWER
I20260923 23:05:23.292706 7656 raft_consensus.cc:348] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.292739 7657 consensus_queue.cc:260] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.292781 7656 raft_consensus.cc:374] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:23.292821 7656 raft_consensus.cc:729] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8770c69dfafd4992bf0fb87067f162af, State: Initialized, Role: FOLLOWER
I20260923 23:05:23.292837 7657 ts_tablet_manager.cc:1428] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:23.292876 7656 consensus_queue.cc:260] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.292992 7656 ts_tablet_manager.cc:1428] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:23.292973 7661 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.293035 7661 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:23.292137 7654 raft_consensus.cc:729] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63040bc26b54b60baf84b93f0aab814, State: Initialized, Role: FOLLOWER
I20260923 23:05:23.293136 7662 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.293215 7662 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:23.293231 7654 consensus_queue.cc:260] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.293344 7654 ts_tablet_manager.cc:1428] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 23:05:23.293422 7658 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.293511 7658 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 23:05:23.348110 7632 tablet.cc:2369] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:23.408541 7661 raft_consensus.cc:482] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:23.408663 7661 raft_consensus.cc:504] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.408830 7661 leader_election.cc:288] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717), d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643)
I20260923 23:05:23.409083 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a738ffee247e4d1ab1473f1f548259a1" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770c69dfafd4992bf0fb87067f162af" is_pre_election: true
I20260923 23:05:23.409189 7305 raft_consensus.cc:2454] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 0.
I20260923 23:05:23.409335 7379 leader_election.cc:302] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0626a976514613829699d7d07d0adf, 8770c69dfafd4992bf0fb87067f162af; no voters:
I20260923 23:05:23.409422 7661 raft_consensus.cc:2781] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:23.409363 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a738ffee247e4d1ab1473f1f548259a1" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814" is_pre_election: true
I20260923 23:05:23.409461 7661 raft_consensus.cc:482] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:23.409482 7661 raft_consensus.cc:3037] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:23.409484 7585 raft_consensus.cc:2454] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 0.
I20260923 23:05:23.410488 7661 raft_consensus.cc:504] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.410657 7661 leader_election.cc:288] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 election: Requested vote from peers 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717), d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643)
I20260923 23:05:23.410828 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a738ffee247e4d1ab1473f1f548259a1" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770c69dfafd4992bf0fb87067f162af"
I20260923 23:05:23.410836 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a738ffee247e4d1ab1473f1f548259a1" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814"
I20260923 23:05:23.410912 7305 raft_consensus.cc:3037] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:23.410938 7585 raft_consensus.cc:3037] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:23.411546 7305 raft_consensus.cc:2454] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 1.
I20260923 23:05:23.411546 7585 raft_consensus.cc:2454] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 1.
I20260923 23:05:23.411741 7379 leader_election.cc:302] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0626a976514613829699d7d07d0adf, 8770c69dfafd4992bf0fb87067f162af; no voters:
I20260923 23:05:23.411854 7661 raft_consensus.cc:2781] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:23.411926 7661 raft_consensus.cc:686] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [term 1 LEADER]: Becoming Leader. State: Replica: 1f0626a976514613829699d7d07d0adf, State: Running, Role: LEADER
I20260923 23:05:23.412002 7661 consensus_queue.cc:237] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.412356 7664 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: TxnStatusTablet state changed. Reason: New leader 1f0626a976514613829699d7d07d0adf. Latest consensus state: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.412456 7664 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:05:23.412756 7676 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 23:05:23.412882 7676 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 23:05:23.412832 7146 catalog_manager.cc:5495] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0626a976514613829699d7d07d0adf (127.5.74.2). New cstate: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:23.442643 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.440060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.5.74.2:38761)
0923 23:05:23.440121 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.440124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.440133 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:23.440244 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.440245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.440266 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.440326 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:23.440523 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.441038 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.441830 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.442031 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.442119 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.442307 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:23.442421 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:23.442516 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.442539 (+ 23us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.442560 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:05:23.442652 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.439999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:38761 (local address 127.0.0.1:45352)
0923 23:05:23.440086 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.440098 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.440149 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.440398 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.440399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.440401 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:23.440454 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.440456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.441127 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.441128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.441713 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.441714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.442120 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.442121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.442187 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.442204 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:23.442488 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:23.442497 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.442537 (+ 40us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.442584 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 23:05:23.444569 7585 raft_consensus.cc:1261] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Refusing update from remote peer 1f0626a976514613829699d7d07d0adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:23.444568 7305 raft_consensus.cc:1261] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Refusing update from remote peer 1f0626a976514613829699d7d07d0adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:23.444932 7664 consensus_queue.cc:1035] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:23.445113 7661 consensus_queue.cc:1035] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:23.446331 7662 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: TxnStatusTablet state changed. Reason: New leader 1f0626a976514613829699d7d07d0adf. Latest consensus state: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.446431 7662 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:23.446410 7658 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: TxnStatusTablet state changed. Reason: New leader 1f0626a976514613829699d7d07d0adf. Latest consensus state: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.446521 7658 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:23.447285 7664 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.447376 7664 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:05:23.448546 7664 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.448655 7664 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P 1f0626a976514613829699d7d07d0adf: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:05:23.448655 7658 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.448738 7658 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P d63040bc26b54b60baf84b93f0aab814: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:23.449051 7662 tablet_replica.cc:439] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } } }
I20260923 23:05:23.449149 7662 tablet_replica.cc:442] T a738ffee247e4d1ab1473f1f548259a1 P 8770c69dfafd4992bf0fb87067f162af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:23.449787 7681 mvcc.cc:205] Tried to move back new op lower bound from 7332678547225980928 to 7332678547096076288. Current Snapshot: MvccSnapshot[applied={T|T < 7332678547225980928 or (T in {7332678547225980928})}]
I20260923 23:05:23.450232 7680 mvcc.cc:205] Tried to move back new op lower bound from 7332678547225980928 to 7332678547096076288. Current Snapshot: MvccSnapshot[applied={T|T < 7332678547225980928}]
I20260923 23:05:23.451892 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.449395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.5.74.2:38761)
0923 23:05:23.449431 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.449433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.449440 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:05:23.449464 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.449465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.449474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.449519 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:23.449687 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.450379 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.451213 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.451431 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.451510 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.451663 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:23.451694 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:23.451738 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.451800 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.451819 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:05:23.457495 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.454434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:36773 (local address 127.5.74.2:38761)
0923 23:05:23.454474 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:23.454476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:23.454642 (+ 166us) server_negotiation.cc:407] Connection header received
0923 23:05:23.454684 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:23.454685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:23.454696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:23.454761 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:23.454877 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.455655 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.456616 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.456944 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.457023 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.457199 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:23.457233 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:23.457297 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:23.457350 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 23:05:23.457369 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:05:23.457504 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:23.454423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:38761 (local address 127.5.74.2:36773)
0923 23:05:23.454564 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:23.454576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:23.454634 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:23.454763 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:23.454763 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:23.454764 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:23.454819 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.454820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.455750 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.455751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.456536 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:23.456538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:23.457039 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:23.457040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:23.457089 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:23.457102 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:23.457301 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:23.457303 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:05:23.457335 (+ 32us) client_negotiation.cc:240] Negotiation successful
0923 23:05:23.457407 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":78,"threads_started":1}
I20260923 23:05:23.460319 7305 raft_consensus.cc:1261] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Refusing update from remote peer 1f0626a976514613829699d7d07d0adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:23.460392 7585 raft_consensus.cc:1261] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Refusing update from remote peer 1f0626a976514613829699d7d07d0adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:23.460544 7691 consensus_queue.cc:1035] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:23.460683 7661 consensus_queue.cc:1035] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:23.475945 7147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:23.476070 7147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:23.476908 7147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48: Sending DeleteTablet for 3 replicas of tablet 4232bd203eb942c7a20326c86a53ce22
I20260923 23:05:23.477210 7565 tablet_service.cc:1514] Processing DeleteTablet for tablet 4232bd203eb942c7a20326c86a53ce22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:23 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:56038
I20260923 23:05:23.477319 7285 tablet_service.cc:1514] Processing DeleteTablet for tablet 4232bd203eb942c7a20326c86a53ce22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:23 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:57118
I20260923 23:05:23.477535 7696 tablet_replica.cc:330] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: stopping tablet replica
I20260923 23:05:23.477520 7147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:23.477662 7147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:23.477685 7696 raft_consensus.cc:2229] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:23.477752 7696 raft_consensus.cc:2258] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:23.478030 7697 tablet_replica.cc:330] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: stopping tablet replica
I20260923 23:05:23.477985 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet 4232bd203eb942c7a20326c86a53ce22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:23 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:45336
I20260923 23:05:23.478144 7697 raft_consensus.cc:2229] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:23.478205 7697 raft_consensus.cc:2258] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:23.478678 7697 ts_tablet_manager.cc:1905] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:23.478973 7565 tablet_service.cc:1467] Processing CreateTablet for tablet 21a4535d2adc42d496a43a3a424f1fc2 (DEFAULT_TABLE table=test-table [id=725bd7d9a1a24d8eb79dabc627abf882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:23.479031 7285 tablet_service.cc:1467] Processing CreateTablet for tablet 21a4535d2adc42d496a43a3a424f1fc2 (DEFAULT_TABLE table=test-table [id=725bd7d9a1a24d8eb79dabc627abf882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:23.479079 7565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a4535d2adc42d496a43a3a424f1fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:23.479138 7285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a4535d2adc42d496a43a3a424f1fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:23.480415 7654 tablet_bootstrap.cc:492] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: Bootstrap starting.
I20260923 23:05:23.480616 7697 ts_tablet_manager.cc:1918] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:23.480685 7697 log.cc:1196] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/4232bd203eb942c7a20326c86a53ce22
I20260923 23:05:23.480756 7656 tablet_bootstrap.cc:492] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: Bootstrap starting.
I20260923 23:05:23.480856 7654 tablet_bootstrap.cc:654] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:23.480968 7697 ts_tablet_manager.cc:1939] T 4232bd203eb942c7a20326c86a53ce22 P 8770c69dfafd4992bf0fb87067f162af: Deleting consensus metadata
I20260923 23:05:23.481138 7656 tablet_bootstrap.cc:654] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:23.481520 7135 catalog_manager.cc:4841] TS 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717): tablet 4232bd203eb942c7a20326c86a53ce22 (table test-table [id=d410ec68429942ecb8d7469373f59255]) successfully deleted
I20260923 23:05:23.481738 7424 tablet_service.cc:1467] Processing CreateTablet for tablet 21a4535d2adc42d496a43a3a424f1fc2 (DEFAULT_TABLE table=test-table [id=725bd7d9a1a24d8eb79dabc627abf882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:23.481866 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a4535d2adc42d496a43a3a424f1fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:23.481966 7696 ts_tablet_manager.cc:1905] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:23.482095 7656 tablet_bootstrap.cc:492] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: No bootstrap required, opened a new log
I20260923 23:05:23.482136 7656 ts_tablet_manager.cc:1397] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:23.482260 7656 raft_consensus.cc:348] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.482322 7656 raft_consensus.cc:374] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:23.482342 7656 raft_consensus.cc:729] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8770c69dfafd4992bf0fb87067f162af, State: Initialized, Role: FOLLOWER
I20260923 23:05:23.482381 7656 consensus_queue.cc:260] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.482457 7656 ts_tablet_manager.cc:1428] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:23.483022 7698 tablet_replica.cc:330] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: stopping tablet replica
I20260923 23:05:23.483134 7698 raft_consensus.cc:2229] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:23.483229 7698 raft_consensus.cc:2258] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:23.483326 7657 tablet_bootstrap.cc:492] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: Bootstrap starting.
I20260923 23:05:23.483594 7696 ts_tablet_manager.cc:1918] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:23.483654 7696 log.cc:1196] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/4232bd203eb942c7a20326c86a53ce22
I20260923 23:05:23.483647 7698 ts_tablet_manager.cc:1905] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:23.483732 7657 tablet_bootstrap.cc:654] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:23.483947 7696 ts_tablet_manager.cc:1939] T 4232bd203eb942c7a20326c86a53ce22 P d63040bc26b54b60baf84b93f0aab814: Deleting consensus metadata
I20260923 23:05:23.484602 7657 tablet_bootstrap.cc:492] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: No bootstrap required, opened a new log
I20260923 23:05:23.484593 7134 catalog_manager.cc:4841] TS d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643): tablet 4232bd203eb942c7a20326c86a53ce22 (table test-table [id=d410ec68429942ecb8d7469373f59255]) successfully deleted
I20260923 23:05:23.484650 7657 ts_tablet_manager.cc:1397] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:23.484786 7657 raft_consensus.cc:348] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.484836 7657 raft_consensus.cc:374] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:23.484850 7657 raft_consensus.cc:729] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0626a976514613829699d7d07d0adf, State: Initialized, Role: FOLLOWER
I20260923 23:05:23.484915 7657 consensus_queue.cc:260] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.485000 7657 ts_tablet_manager.cc:1428] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:23.485411 7654 tablet_bootstrap.cc:492] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: No bootstrap required, opened a new log
I20260923 23:05:23.485474 7654 ts_tablet_manager.cc:1397] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260923 23:05:23.485637 7654 raft_consensus.cc:348] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.485718 7654 raft_consensus.cc:374] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:23.485740 7654 raft_consensus.cc:729] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63040bc26b54b60baf84b93f0aab814, State: Initialized, Role: FOLLOWER
I20260923 23:05:23.485778 7654 consensus_queue.cc:260] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.485883 7654 ts_tablet_manager.cc:1428] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:23.486116 7698 ts_tablet_manager.cc:1918] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:23.486191 7698 log.cc:1196] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/4232bd203eb942c7a20326c86a53ce22
I20260923 23:05:23.486488 7698 ts_tablet_manager.cc:1939] T 4232bd203eb942c7a20326c86a53ce22 P 1f0626a976514613829699d7d07d0adf: Deleting consensus metadata
I20260923 23:05:23.486918 7132 catalog_manager.cc:4841] TS 1f0626a976514613829699d7d07d0adf (127.5.74.2:38761): tablet 4232bd203eb942c7a20326c86a53ce22 (table test-table [id=d410ec68429942ecb8d7469373f59255]) successfully deleted
I20260923 23:05:23.567116 7695 raft_consensus.cc:482] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:23.567353 7695 raft_consensus.cc:504] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.567543 7695 leader_election.cc:288] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717), d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643)
I20260923 23:05:23.567737 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a4535d2adc42d496a43a3a424f1fc2" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770c69dfafd4992bf0fb87067f162af" is_pre_election: true
I20260923 23:05:23.567855 7305 raft_consensus.cc:2454] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 0.
I20260923 23:05:23.567821 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a4535d2adc42d496a43a3a424f1fc2" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814" is_pre_election: true
I20260923 23:05:23.567937 7585 raft_consensus.cc:2454] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 0.
I20260923 23:05:23.568027 7379 leader_election.cc:302] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0626a976514613829699d7d07d0adf, 8770c69dfafd4992bf0fb87067f162af; no voters:
I20260923 23:05:23.568123 7695 raft_consensus.cc:2781] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:23.568184 7695 raft_consensus.cc:482] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:23.568223 7695 raft_consensus.cc:3037] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:23.568945 7695 raft_consensus.cc:504] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.569120 7695 leader_election.cc:288] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 election: Requested vote from peers 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717), d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643)
I20260923 23:05:23.569305 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a4535d2adc42d496a43a3a424f1fc2" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814"
I20260923 23:05:23.569286 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a4535d2adc42d496a43a3a424f1fc2" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770c69dfafd4992bf0fb87067f162af"
I20260923 23:05:23.569402 7305 raft_consensus.cc:3037] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:23.569403 7585 raft_consensus.cc:3037] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:23.570050 7585 raft_consensus.cc:2454] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 1.
I20260923 23:05:23.570202 7305 raft_consensus.cc:2454] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 1.
I20260923 23:05:23.570225 7378 leader_election.cc:302] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0626a976514613829699d7d07d0adf, d63040bc26b54b60baf84b93f0aab814; no voters:
I20260923 23:05:23.570340 7695 raft_consensus.cc:2781] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:23.570384 7695 raft_consensus.cc:686] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 1 LEADER]: Becoming Leader. State: Replica: 1f0626a976514613829699d7d07d0adf, State: Running, Role: LEADER
I20260923 23:05:23.570443 7695 consensus_queue.cc:237] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:23.571066 7147 catalog_manager.cc:5495] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0626a976514613829699d7d07d0adf (127.5.74.2). New cstate: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:23.580863 7585 raft_consensus.cc:1261] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Refusing update from remote peer 1f0626a976514613829699d7d07d0adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:23.580926 7305 raft_consensus.cc:1261] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Refusing update from remote peer 1f0626a976514613829699d7d07d0adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:23.581120 7695 consensus_queue.cc:1035] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:23.581265 7661 consensus_queue.cc:1035] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:23.582228 7679 mvcc.cc:205] Tried to move back new op lower bound from 7332678547785527296 to 7332678547744894976. Current Snapshot: MvccSnapshot[applied={T|T < 7332678547785527296}]
W20260923 23:05:23.585675 7679 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 23:05:25.106165 7146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:25.106281 7146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:25.107398 7146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48: Sending DeleteTablet for 3 replicas of tablet 21a4535d2adc42d496a43a3a424f1fc2
I20260923 23:05:25.107945 7565 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a4535d2adc42d496a43a3a424f1fc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:25 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:56038
I20260923 23:05:25.108050 7285 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a4535d2adc42d496a43a3a424f1fc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:25 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:57118
I20260923 23:05:25.108179 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a4535d2adc42d496a43a3a424f1fc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:25 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:45336
I20260923 23:05:25.108258 7725 tablet_replica.cc:330] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: stopping tablet replica
I20260923 23:05:25.108434 7726 tablet_replica.cc:330] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: stopping tablet replica
I20260923 23:05:25.108489 7726 raft_consensus.cc:2229] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:25.108517 7726 raft_consensus.cc:2258] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:25.108618 7725 raft_consensus.cc:2229] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:25.108693 7725 raft_consensus.cc:2258] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:25.108937 7725 ts_tablet_manager.cc:1905] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:25.109009 7726 ts_tablet_manager.cc:1905] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:25.110625 7727 tablet_replica.cc:330] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: stopping tablet replica
I20260923 23:05:25.110730 7727 raft_consensus.cc:2229] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:25.110850 7727 raft_consensus.cc:2258] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:25.110912 7726 ts_tablet_manager.cc:1918] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:05:25.110970 7726 log.cc:1196] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/21a4535d2adc42d496a43a3a424f1fc2
I20260923 23:05:25.111227 7726 ts_tablet_manager.cc:1939] T 21a4535d2adc42d496a43a3a424f1fc2 P 8770c69dfafd4992bf0fb87067f162af: Deleting consensus metadata
I20260923 23:05:25.111228 7727 ts_tablet_manager.cc:1905] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:25.111407 7725 ts_tablet_manager.cc:1918] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:05:25.111490 7725 log.cc:1196] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/21a4535d2adc42d496a43a3a424f1fc2
I20260923 23:05:25.111722 7725 ts_tablet_manager.cc:1939] T 21a4535d2adc42d496a43a3a424f1fc2 P d63040bc26b54b60baf84b93f0aab814: Deleting consensus metadata
I20260923 23:05:25.111791 7135 catalog_manager.cc:4841] TS 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717): tablet 21a4535d2adc42d496a43a3a424f1fc2 (table test-table [id=725bd7d9a1a24d8eb79dabc627abf882]) successfully deleted
I20260923 23:05:25.112064 7134 catalog_manager.cc:4841] TS d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643): tablet 21a4535d2adc42d496a43a3a424f1fc2 (table test-table [id=725bd7d9a1a24d8eb79dabc627abf882]) successfully deleted
I20260923 23:05:25.113867 7727 ts_tablet_manager.cc:1918] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:05:25.113957 7727 log.cc:1196] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/21a4535d2adc42d496a43a3a424f1fc2
I20260923 23:05:25.114245 7727 ts_tablet_manager.cc:1939] T 21a4535d2adc42d496a43a3a424f1fc2 P 1f0626a976514613829699d7d07d0adf: Deleting consensus metadata
I20260923 23:05:25.114766 7132 catalog_manager.cc:4841] TS 1f0626a976514613829699d7d07d0adf (127.5.74.2:38761): tablet 21a4535d2adc42d496a43a3a424f1fc2 (table test-table [id=725bd7d9a1a24d8eb79dabc627abf882]) successfully deleted
Sep 23 23:05:25 dist-test-slave-j4c5 krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:05:25 dist-test-slave-j4c5 krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:25.143530 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.131217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.5.74.62:36053)
0923 23:05:25.131346 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:25.131349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:25.131375 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:05:25.131405 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:25.131405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:25.131419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:25.131495 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:25.132599 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.133669 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.134950 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.135188 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.135289 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.139472 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:25.139474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:25.139474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:25.139484 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:25.141785 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:25.142328 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:25.142330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:25.142330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:25.142407 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:25.142684 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:25.142685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:25.142686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:25.142864 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:25.143003 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:25.143369 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 23:05:25.143447 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260923 23:05:25.145610 7146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:25.145818 7146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:25.147408 7565 tablet_service.cc:1467] Processing CreateTablet for tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb (DEFAULT_TABLE table=test-table [id=dc098f206ec748bb8c9d279ecd24c22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:25.147400 7285 tablet_service.cc:1467] Processing CreateTablet for tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb (DEFAULT_TABLE table=test-table [id=dc098f206ec748bb8c9d279ecd24c22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:25.147542 7285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:25.147646 7565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:25.147646 7424 tablet_service.cc:1467] Processing CreateTablet for tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb (DEFAULT_TABLE table=test-table [id=dc098f206ec748bb8c9d279ecd24c22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:25.147796 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:25.149694 7737 tablet_bootstrap.cc:492] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: Bootstrap starting.
I20260923 23:05:25.149900 7738 tablet_bootstrap.cc:492] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: Bootstrap starting.
I20260923 23:05:25.150134 7737 tablet_bootstrap.cc:654] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:25.150249 7738 tablet_bootstrap.cc:654] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:25.150478 7736 tablet_bootstrap.cc:492] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: Bootstrap starting.
I20260923 23:05:25.150823 7737 tablet_bootstrap.cc:492] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: No bootstrap required, opened a new log
I20260923 23:05:25.150882 7737 ts_tablet_manager.cc:1397] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:25.150898 7736 tablet_bootstrap.cc:654] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:25.151062 7737 raft_consensus.cc:348] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.151160 7737 raft_consensus.cc:374] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:25.151185 7737 raft_consensus.cc:729] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8770c69dfafd4992bf0fb87067f162af, State: Initialized, Role: FOLLOWER
I20260923 23:05:25.151424 7738 tablet_bootstrap.cc:492] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: No bootstrap required, opened a new log
I20260923 23:05:25.151479 7738 ts_tablet_manager.cc:1397] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:25.151588 7738 raft_consensus.cc:348] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.151659 7738 raft_consensus.cc:374] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:25.151681 7738 raft_consensus.cc:729] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0626a976514613829699d7d07d0adf, State: Initialized, Role: FOLLOWER
I20260923 23:05:25.151736 7738 consensus_queue.cc:260] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.151845 7736 tablet_bootstrap.cc:492] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: No bootstrap required, opened a new log
I20260923 23:05:25.151847 7738 ts_tablet_manager.cc:1428] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:25.151893 7736 ts_tablet_manager.cc:1397] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:25.151923 7737 consensus_queue.cc:260] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.152030 7736 raft_consensus.cc:348] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.152113 7736 raft_consensus.cc:374] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:25.152141 7736 raft_consensus.cc:729] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63040bc26b54b60baf84b93f0aab814, State: Initialized, Role: FOLLOWER
I20260923 23:05:25.152132 7737 ts_tablet_manager.cc:1428] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:25.152225 7736 consensus_queue.cc:260] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.152433 7736 ts_tablet_manager.cc:1428] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:25.204696 7742 raft_consensus.cc:482] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:25.204813 7742 raft_consensus.cc:504] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.205215 7742 leader_election.cc:288] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0626a976514613829699d7d07d0adf (127.5.74.2:38761), d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643)
I20260923 23:05:25.210033 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.205492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:49871 (local address 127.5.74.3:33643)
0923 23:05:25.205673 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:25.205675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:25.205801 (+ 126us) server_negotiation.cc:407] Connection header received
0923 23:05:25.205849 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:25.205850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:25.205889 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:25.205979 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:25.206273 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.207088 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.208763 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.209507 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.209566 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.209577 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:25.209918 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 23:05:25.209966 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
I20260923 23:05:25.210054 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.205418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:33643 (local address 127.5.74.1:49871)
0923 23:05:25.205703 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:25.205716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:25.205788 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:25.206034 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:25.206036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:25.206039 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:25.206138 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:25.206140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.207211 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.207212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:25.208653 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:25.208658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.209735 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.209736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:25.209820 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.209825 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:25.209902 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 23:05:25.209951 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":26,"thread_start_us":194,"threads_started":1}
I20260923 23:05:25.210453 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f8bb1d903f43d9bc4e3e52ec3ba8cb" candidate_uuid: "8770c69dfafd4992bf0fb87067f162af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814" is_pre_election: true
I20260923 23:05:25.210650 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.205539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:38761 (local address 127.5.74.1:56557)
0923 23:05:25.205732 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:25.205742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:25.205788 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:25.206035 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:25.206036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:25.206040 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:25.206138 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:25.206143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.207218 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.207219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:25.209236 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:25.209240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.210216 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.210217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:25.210292 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.210297 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:25.210355 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:05:25.210403 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":86,"thread_start_us":69,"threads_started":1}
I20260923 23:05:25.210736 7585 raft_consensus.cc:2454] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8770c69dfafd4992bf0fb87067f162af in term 0.
I20260923 23:05:25.210908 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.205647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:56557 (local address 127.5.74.2:38761)
0923 23:05:25.205810 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:25.205812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:25.205821 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:25.205874 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:25.205874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:25.205887 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:25.205979 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:25.206274 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.207101 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.209393 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.210081 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.210172 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.210184 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:25.210785 (+ 601us) server_negotiation.cc:299] Negotiation successful
0923 23:05:25.210844 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":107,"threads_started":1}
I20260923 23:05:25.211025 7238 leader_election.cc:302] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8770c69dfafd4992bf0fb87067f162af, d63040bc26b54b60baf84b93f0aab814; no voters:
I20260923 23:05:25.211182 7742 raft_consensus.cc:2781] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:25.211218 7742 raft_consensus.cc:482] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:25.211251 7742 raft_consensus.cc:3037] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:25.211189 7445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f8bb1d903f43d9bc4e3e52ec3ba8cb" candidate_uuid: "8770c69dfafd4992bf0fb87067f162af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0626a976514613829699d7d07d0adf" is_pre_election: true
I20260923 23:05:25.211328 7445 raft_consensus.cc:2454] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8770c69dfafd4992bf0fb87067f162af in term 0.
I20260923 23:05:25.212040 7742 raft_consensus.cc:504] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.212234 7742 leader_election.cc:288] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [CANDIDATE]: Term 1 election: Requested vote from peers 1f0626a976514613829699d7d07d0adf (127.5.74.2:38761), d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643)
I20260923 23:05:25.212387 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f8bb1d903f43d9bc4e3e52ec3ba8cb" candidate_uuid: "8770c69dfafd4992bf0fb87067f162af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814"
I20260923 23:05:25.212381 7445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f8bb1d903f43d9bc4e3e52ec3ba8cb" candidate_uuid: "8770c69dfafd4992bf0fb87067f162af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0626a976514613829699d7d07d0adf"
I20260923 23:05:25.212486 7585 raft_consensus.cc:3037] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:25.212484 7445 raft_consensus.cc:3037] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:25.213330 7585 raft_consensus.cc:2454] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8770c69dfafd4992bf0fb87067f162af in term 1.
I20260923 23:05:25.213384 7445 raft_consensus.cc:2454] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8770c69dfafd4992bf0fb87067f162af in term 1.
I20260923 23:05:25.213577 7238 leader_election.cc:302] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8770c69dfafd4992bf0fb87067f162af, d63040bc26b54b60baf84b93f0aab814; no voters:
I20260923 23:05:25.213706 7742 raft_consensus.cc:2781] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:25.213892 7742 raft_consensus.cc:686] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 1 LEADER]: Becoming Leader. State: Replica: 8770c69dfafd4992bf0fb87067f162af, State: Running, Role: LEADER
I20260923 23:05:25.214012 7742 consensus_queue.cc:237] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.214973 7147 catalog_manager.cc:5495] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af reported cstate change: term changed from 0 to 1, leader changed from <none> to 8770c69dfafd4992bf0fb87067f162af (127.5.74.1). New cstate: current_term: 1 leader_uuid: "8770c69dfafd4992bf0fb87067f162af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:25.232450 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.229045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.5.74.1:44717)
0923 23:05:25.229391 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:25.229393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:25.229411 (+ 18us) server_negotiation.cc:407] Connection header received
0923 23:05:25.229448 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:25.229449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:25.229468 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:25.229558 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:25.229774 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.230290 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.231284 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.231499 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.231606 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.231991 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:25.232112 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:25.232252 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:25.232327 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 23:05:25.232348 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":199,"threads_started":1}
I20260923 23:05:25.238142 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.235290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:41619 (local address 127.5.74.2:38761)
0923 23:05:25.235426 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:25.235428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:25.235566 (+ 138us) server_negotiation.cc:407] Connection header received
0923 23:05:25.235595 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:25.235596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:25.235607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:25.235664 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:25.235794 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.236435 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.237283 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.237521 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.237629 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.237856 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:25.237898 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:25.237958 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:25.238028 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 23:05:25.238052 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260923 23:05:25.238154 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.235206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:38761 (local address 127.5.74.1:41619)
0923 23:05:25.235429 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:25.235440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:25.235513 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:25.235675 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:25.235676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:25.235677 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:25.235739 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:25.235740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.236560 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.236561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:25.237185 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:25.237187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.237639 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.237640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:25.237721 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.237737 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:25.237974 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:25.237976 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:05:25.238016 (+ 40us) client_negotiation.cc:240] Negotiation successful
0923 23:05:25.238064 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":160,"threads_started":1}
I20260923 23:05:25.240927 7585 raft_consensus.cc:1261] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Refusing update from remote peer 8770c69dfafd4992bf0fb87067f162af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:25.241041 7445 raft_consensus.cc:1261] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Refusing update from remote peer 8770c69dfafd4992bf0fb87067f162af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:25.241262 7748 consensus_queue.cc:1035] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:25.241384 7742 consensus_queue.cc:1035] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:25.250095 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.246847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:42899 (local address 127.5.74.1:44717)
0923 23:05:25.246893 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:25.246896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:25.246990 (+ 94us) server_negotiation.cc:407] Connection header received
0923 23:05:25.247059 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:25.247060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:25.247075 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:25.247134 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:25.247297 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.248020 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.249316 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.249528 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.249617 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.249843 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:25.249876 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:25.249928 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:25.249990 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 23:05:25.250012 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:05:25.250111 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.246794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:44717 (local address 127.5.74.2:42899)
0923 23:05:25.246924 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:25.246935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:25.247004 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:25.247150 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:25.247151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:25.247153 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:25.247204 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:25.247206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.248126 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.248127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:25.249220 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:25.249224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.249621 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.249622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:25.249679 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.249684 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:25.249936 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:25.249938 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:05:25.249988 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 23:05:25.250034 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260923 23:05:25.260591 7146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:25.260751 7146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:25.261654 7146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48: Sending DeleteTablet for 3 replicas of tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb
I20260923 23:05:25.261998 7565 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:25 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:56038
I20260923 23:05:25.262048 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:25 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:45336
I20260923 23:05:25.262063 7285 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:25 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:57118
I20260923 23:05:25.262112 7725 tablet_replica.cc:330] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: stopping tablet replica
I20260923 23:05:25.262148 7726 tablet_replica.cc:330] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: stopping tablet replica
I20260923 23:05:25.262156 7727 tablet_replica.cc:330] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: stopping tablet replica
I20260923 23:05:25.262166 7725 raft_consensus.cc:2229] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:25.262190 7726 raft_consensus.cc:2229] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:25.262204 7727 raft_consensus.cc:2229] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:25.262213 7725 raft_consensus.cc:2258] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:25.262244 7727 raft_consensus.cc:2258] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:25.262283 7726 raft_consensus.cc:2258] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:25.262575 7725 ts_tablet_manager.cc:1905] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:25.262575 7726 ts_tablet_manager.cc:1905] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:25.264364 7727 ts_tablet_manager.cc:1905] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:25.264525 7146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 23:05:25.264660 7725 ts_tablet_manager.cc:1918] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:25.264709 7725 log.cc:1196] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/25f8bb1d903f43d9bc4e3e52ec3ba8cb
W20260923 23:05:25.264747 7146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:25.264925 7725 ts_tablet_manager.cc:1939] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P d63040bc26b54b60baf84b93f0aab814: Deleting consensus metadata
I20260923 23:05:25.265403 7726 ts_tablet_manager.cc:1918] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:25.265473 7726 log.cc:1196] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/25f8bb1d903f43d9bc4e3e52ec3ba8cb
I20260923 23:05:25.265678 7726 ts_tablet_manager.cc:1939] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 8770c69dfafd4992bf0fb87067f162af: Deleting consensus metadata
I20260923 23:05:25.266048 7135 catalog_manager.cc:4841] TS 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717): tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb (table test-table [id=dc098f206ec748bb8c9d279ecd24c22d]) successfully deleted
I20260923 23:05:25.266173 7134 catalog_manager.cc:4841] TS d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643): tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb (table test-table [id=dc098f206ec748bb8c9d279ecd24c22d]) successfully deleted
I20260923 23:05:25.266171 7727 ts_tablet_manager.cc:1918] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:25.266232 7727 log.cc:1196] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/25f8bb1d903f43d9bc4e3e52ec3ba8cb
I20260923 23:05:25.266459 7727 ts_tablet_manager.cc:1939] T 25f8bb1d903f43d9bc4e3e52ec3ba8cb P 1f0626a976514613829699d7d07d0adf: Deleting consensus metadata
I20260923 23:05:25.267228 7132 catalog_manager.cc:4841] TS 1f0626a976514613829699d7d07d0adf (127.5.74.2:38761): tablet 25f8bb1d903f43d9bc4e3e52ec3ba8cb (table test-table [id=dc098f206ec748bb8c9d279ecd24c22d]) successfully deleted
I20260923 23:05:25.267522 7285 tablet_service.cc:1467] Processing CreateTablet for tablet eddc276941524c86b3d9ad7b39e64609 (DEFAULT_TABLE table=test-table [id=b2aff2ebd17645fe8191250a0bd35766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:25.267557 7565 tablet_service.cc:1467] Processing CreateTablet for tablet eddc276941524c86b3d9ad7b39e64609 (DEFAULT_TABLE table=test-table [id=b2aff2ebd17645fe8191250a0bd35766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:25.267648 7285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eddc276941524c86b3d9ad7b39e64609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:25.267674 7565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eddc276941524c86b3d9ad7b39e64609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:25.268994 7737 tablet_bootstrap.cc:492] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af: Bootstrap starting.
I20260923 23:05:25.269017 7736 tablet_bootstrap.cc:492] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: Bootstrap starting.
I20260923 23:05:25.269039 7424 tablet_service.cc:1467] Processing CreateTablet for tablet eddc276941524c86b3d9ad7b39e64609 (DEFAULT_TABLE table=test-table [id=b2aff2ebd17645fe8191250a0bd35766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:25.269146 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eddc276941524c86b3d9ad7b39e64609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:25.269351 7736 tablet_bootstrap.cc:654] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:25.269413 7737 tablet_bootstrap.cc:654] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:25.270303 7736 tablet_bootstrap.cc:492] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: No bootstrap required, opened a new log
I20260923 23:05:25.270362 7736 ts_tablet_manager.cc:1397] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:25.270466 7737 tablet_bootstrap.cc:492] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af: No bootstrap required, opened a new log
I20260923 23:05:25.270504 7737 ts_tablet_manager.cc:1397] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:25.270527 7736 raft_consensus.cc:348] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.270612 7736 raft_consensus.cc:374] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:25.270633 7736 raft_consensus.cc:729] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63040bc26b54b60baf84b93f0aab814, State: Initialized, Role: FOLLOWER
I20260923 23:05:25.270658 7737 raft_consensus.cc:348] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.270682 7736 consensus_queue.cc:260] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.270717 7737 raft_consensus.cc:374] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:25.270740 7737 raft_consensus.cc:729] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8770c69dfafd4992bf0fb87067f162af, State: Initialized, Role: FOLLOWER
I20260923 23:05:25.270773 7736 ts_tablet_manager.cc:1428] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:25.270780 7737 consensus_queue.cc:260] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.270887 7737 ts_tablet_manager.cc:1428] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:25.271016 7738 tablet_bootstrap.cc:492] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: Bootstrap starting.
I20260923 23:05:25.271376 7738 tablet_bootstrap.cc:654] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:25.272128 7738 tablet_bootstrap.cc:492] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: No bootstrap required, opened a new log
I20260923 23:05:25.272202 7738 ts_tablet_manager.cc:1397] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:25.272311 7738 raft_consensus.cc:348] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.272383 7738 raft_consensus.cc:374] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:25.272401 7738 raft_consensus.cc:729] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0626a976514613829699d7d07d0adf, State: Initialized, Role: FOLLOWER
I20260923 23:05:25.272446 7738 consensus_queue.cc:260] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.272548 7738 ts_tablet_manager.cc:1428] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:25.293612 7708 raft_consensus.cc:482] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:25.293709 7708 raft_consensus.cc:504] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.293946 7708 leader_election.cc:288] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717), d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643)
I20260923 23:05:25.294135 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eddc276941524c86b3d9ad7b39e64609" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814" is_pre_election: true
I20260923 23:05:25.294162 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eddc276941524c86b3d9ad7b39e64609" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770c69dfafd4992bf0fb87067f162af" is_pre_election: true
I20260923 23:05:25.294237 7585 raft_consensus.cc:2454] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 0.
I20260923 23:05:25.294256 7305 raft_consensus.cc:2454] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 0.
I20260923 23:05:25.294412 7378 leader_election.cc:302] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0626a976514613829699d7d07d0adf, d63040bc26b54b60baf84b93f0aab814; no voters:
I20260923 23:05:25.294508 7708 raft_consensus.cc:2781] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:25.294545 7708 raft_consensus.cc:482] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:25.294566 7708 raft_consensus.cc:3037] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:25.295276 7708 raft_consensus.cc:504] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.295428 7708 leader_election.cc:288] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 election: Requested vote from peers 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717), d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643)
I20260923 23:05:25.295558 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eddc276941524c86b3d9ad7b39e64609" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770c69dfafd4992bf0fb87067f162af"
I20260923 23:05:25.295639 7305 raft_consensus.cc:3037] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:25.295614 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eddc276941524c86b3d9ad7b39e64609" candidate_uuid: "1f0626a976514613829699d7d07d0adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63040bc26b54b60baf84b93f0aab814"
I20260923 23:05:25.295678 7585 raft_consensus.cc:3037] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:25.296485 7585 raft_consensus.cc:2454] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 1.
I20260923 23:05:25.296479 7305 raft_consensus.cc:2454] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0626a976514613829699d7d07d0adf in term 1.
I20260923 23:05:25.296691 7378 leader_election.cc:302] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0626a976514613829699d7d07d0adf, d63040bc26b54b60baf84b93f0aab814; no voters:
I20260923 23:05:25.296825 7708 raft_consensus.cc:2781] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:25.296880 7708 raft_consensus.cc:686] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 1 LEADER]: Becoming Leader. State: Replica: 1f0626a976514613829699d7d07d0adf, State: Running, Role: LEADER
I20260923 23:05:25.296939 7708 consensus_queue.cc:237] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } }
I20260923 23:05:25.297593 7147 catalog_manager.cc:5495] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0626a976514613829699d7d07d0adf (127.5.74.2). New cstate: current_term: 1 leader_uuid: "1f0626a976514613829699d7d07d0adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0626a976514613829699d7d07d0adf" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:25.305038 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:25.302558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.5.74.2:38761)
0923 23:05:25.302606 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:25.302608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:25.302617 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:25.302677 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:25.302678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:25.302688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:25.302741 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:25.302841 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.303459 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.304236 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:25.304522 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:25.304623 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:25.304776 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:25.304810 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:25.304861 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:25.304926 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 23:05:25.304948 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:05:25.307618 7584 raft_consensus.cc:1261] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Refusing update from remote peer 1f0626a976514613829699d7d07d0adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:25.307641 7305 raft_consensus.cc:1261] T eddc276941524c86b3d9ad7b39e64609 P 8770c69dfafd4992bf0fb87067f162af [term 1 FOLLOWER]: Refusing update from remote peer 1f0626a976514613829699d7d07d0adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:25.307900 7708 consensus_queue.cc:1035] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63040bc26b54b60baf84b93f0aab814" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:25.307989 7705 consensus_queue.cc:1035] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8770c69dfafd4992bf0fb87067f162af" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 23:05:25.312125 7706 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 23:05:26.831725 7146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:26.831837 7146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:26.832912 7146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 285f7200162c44f889d11cdca9f1ca48: Sending DeleteTablet for 3 replicas of tablet eddc276941524c86b3d9ad7b39e64609
I20260923 23:05:26.833271 7285 tablet_service.cc:1514] Processing DeleteTablet for tablet eddc276941524c86b3d9ad7b39e64609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:26 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:57118
I20260923 23:05:26.833278 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet eddc276941524c86b3d9ad7b39e64609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:26 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:45336
I20260923 23:05:26.833489 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 7220
I20260923 23:05:26.833503 7565 tablet_service.cc:1514] Processing DeleteTablet for tablet eddc276941524c86b3d9ad7b39e64609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:26 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:56038
I20260923 23:05:26.835886 7789 tablet_replica.cc:330] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: stopping tablet replica
I20260923 23:05:26.835990 7789 raft_consensus.cc:2229] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:26.836045 7789 raft_consensus.cc:2258] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:26.836186 7788 tablet_replica.cc:330] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: stopping tablet replica
I20260923 23:05:26.836257 7788 raft_consensus.cc:2229] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:26.836390 7788 raft_consensus.cc:2258] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:26.836397 7789 ts_tablet_manager.cc:1905] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:26.836659 7788 ts_tablet_manager.cc:1905] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:26.838223 7789 ts_tablet_manager.cc:1918] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:05:26.838359 7789 log.cc:1196] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/eddc276941524c86b3d9ad7b39e64609
I20260923 23:05:26.838603 7789 ts_tablet_manager.cc:1939] T eddc276941524c86b3d9ad7b39e64609 P d63040bc26b54b60baf84b93f0aab814: Deleting consensus metadata
I20260923 23:05:26.838619 7788 ts_tablet_manager.cc:1918] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:05:26.838675 7788 log.cc:1196] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/eddc276941524c86b3d9ad7b39e64609
I20260923 23:05:26.838908 7788 ts_tablet_manager.cc:1939] T eddc276941524c86b3d9ad7b39e64609 P 1f0626a976514613829699d7d07d0adf: Deleting consensus metadata
I20260923 23:05:26.839241 7134 catalog_manager.cc:4841] TS d63040bc26b54b60baf84b93f0aab814 (127.5.74.3:33643): tablet eddc276941524c86b3d9ad7b39e64609 (table test-table [id=b2aff2ebd17645fe8191250a0bd35766]) successfully deleted
I20260923 23:05:26.839640 7132 catalog_manager.cc:4841] TS 1f0626a976514613829699d7d07d0adf (127.5.74.2:38761): tablet eddc276941524c86b3d9ad7b39e64609 (table test-table [id=b2aff2ebd17645fe8191250a0bd35766]) successfully deleted
W20260923 23:05:26.840971 7135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:44717 (error 108)
I20260923 23:05:26.841629 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 7360
I20260923 23:05:26.841648 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:26.841432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:44717 (local address 127.0.0.1:57148)
0923 23:05:26.841565 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:26.841615 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
W20260923 23:05:26.842594 7135 catalog_manager.cc:4584] TS 8770c69dfafd4992bf0fb87067f162af (127.5.74.1:44717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eddc276941524c86b3d9ad7b39e64609: Network error: Client connection negotiation failed: client connection to 127.5.74.1:44717: connect: Connection refused (error 111)
I20260923 23:05:26.848533 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 7500
I20260923 23:05:26.854055 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 7116
2026-09-23T23:05:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5701 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:26 dist-test-slave-j4c5 krb5kdc[7794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:26 dist-test-slave-j4c5 krb5kdc[7794](info): set up 2 sockets
Sep 23 23:05:26 dist-test-slave-j4c5 krb5kdc[7794](info): commencing operation
krb5kdc: starting...
W20260923 23:05:27.984115 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[7794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:28.119670 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:33423
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:33423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:28.209203 7807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:28.209496 7807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:28.209524 7807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:28.210896 7807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:28.210949 7807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:28.210964 7807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:28.210978 7807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:28.210990 7807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:28.212725 7807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:33423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:33423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:28.213013 7807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:28.213272 7807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:28.218868 7813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:28.218868 7812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:28.219105 7815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:28.219256 7807 server_base.cc:1034] running on GCE node
I20260923 23:05:28.219591 7807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:28.219921 7807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:28.221156 7807 hybrid_clock.cc:648] HybridClock initialized: now 1790204728221076 us; error 54 us; skew 500 ppm
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[7794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204728, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:28.224117 7807 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:28.224676 7807 webserver.cc:460] Webserver started at http://127.5.74.62:34981/ using document root <none> and password file <none>
I20260923 23:05:28.224924 7807 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:28.225030 7807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:28.225183 7807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:28.226311 7807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "dfb15f40af024a30add9a1478c354729"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "d72a7e800252164ecdc0f199a72489b1"
server_key_iv: "8190baac1d541ca84f7e53db415e9896"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.226701 7807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "dfb15f40af024a30add9a1478c354729"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "d72a7e800252164ecdc0f199a72489b1"
server_key_iv: "8190baac1d541ca84f7e53db415e9896"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.228607 7807 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 23:05:28.229662 7822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.229949 7807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:28.230087 7807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "dfb15f40af024a30add9a1478c354729"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "d72a7e800252164ecdc0f199a72489b1"
server_key_iv: "8190baac1d541ca84f7e53db415e9896"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.230189 7807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:28.257747 7807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:28.260541 7807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:28.260725 7807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:28.265328 7807 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:33423
I20260923 23:05:28.265475 7874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:33423 every 8 connection(s)
I20260923 23:05:28.265810 7807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:28.266790 7875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:28.269867 7875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729: Bootstrap starting.
I20260923 23:05:28.270532 7875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:28.270872 7875 log.cc:824] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:28.271863 7875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729: No bootstrap required, opened a new log
I20260923 23:05:28.273645 7875 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb15f40af024a30add9a1478c354729" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33423 } }
I20260923 23:05:28.273808 7875 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:28.273831 7875 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb15f40af024a30add9a1478c354729, State: Initialized, Role: FOLLOWER
I20260923 23:05:28.274003 7875 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb15f40af024a30add9a1478c354729" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33423 } }
I20260923 23:05:28.274106 7875 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:28.274143 7875 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:28.274200 7875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:28.274806 7875 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb15f40af024a30add9a1478c354729" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33423 } }
I20260923 23:05:28.274928 7875 leader_election.cc:302] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfb15f40af024a30add9a1478c354729; no voters:
I20260923 23:05:28.275127 7875 leader_election.cc:288] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:28.275185 7878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:28.275419 7875 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:28.275682 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 7807
I20260923 23:05:28.275707 7878 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [term 1 LEADER]: Becoming Leader. State: Replica: dfb15f40af024a30add9a1478c354729, State: Running, Role: LEADER
I20260923 23:05:28.275775 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:28.275810 7878 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb15f40af024a30add9a1478c354729" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33423 } }
I20260923 23:05:28.276119 5416 external_mini_cluster.cc:1428] Setting key fd0054aa28783c64e7eadbb38d0ea39b
I20260923 23:05:28.276418 7880 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfb15f40af024a30add9a1478c354729. Latest consensus state: current_term: 1 leader_uuid: "dfb15f40af024a30add9a1478c354729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb15f40af024a30add9a1478c354729" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33423 } } }
I20260923 23:05:28.276499 7880 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:28.276670 7879 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfb15f40af024a30add9a1478c354729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb15f40af024a30add9a1478c354729" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33423 } } }
I20260923 23:05:28.277362 7879 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:28.277540 7882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:28.278618 7882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:28.280350 7882 catalog_manager.cc:1347] Generated new cluster ID: 4e3afa8b2976400cb060760204bf0049
I20260923 23:05:28.280407 7882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[7794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:28.290802 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:28.277104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.5.74.62:33423)
0923 23:05:28.277362 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:28.277372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:28.277405 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:05:28.277570 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:28.277578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:28.277636 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:28.277785 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:28.278713 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.280215 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.281322 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.282233 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.282551 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:28.285570 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:28.285577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:28.285578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:28.285597 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:28.287923 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:28.288529 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:28.288533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:28.288534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:28.288684 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:28.289062 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:28.289063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:28.289064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:28.289814 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:28.290059 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:28.290142 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 23:05:28.290458 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20260923 23:05:28.298173 7882 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:28.298789 7882 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:28.308559 7882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfb15f40af024a30add9a1478c354729: Generated new TSK 0
I20260923 23:05:28.308807 7882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:28.354105 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:33423
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:28.455504 7901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:28.455761 7901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:28.455789 7901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:28.457160 7901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:28.457216 7901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:28.457254 7901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:28.458851 7901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:28.459076 7901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:28.459281 7901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:05:28.465221 7901 server_base.cc:1034] running on GCE node
W20260923 23:05:28.465157 7907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:28.465190 7906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:28.465157 7909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:28.465839 7901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:28.466161 7901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:28.467370 7901 hybrid_clock.cc:648] HybridClock initialized: now 1790204728467352 us; error 97 us; skew 500 ppm
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[7794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204728, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:28.471284 7901 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:28.471872 7901 webserver.cc:460] Webserver started at http://127.5.74.1:46125/ using document root <none> and password file <none>
I20260923 23:05:28.472255 7901 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:28.472400 7901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:28.472577 7901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:28.473891 7901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "7c1188c4937a46caa9cf1d14cbcb7779"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "6720e1e03f122d81dc062d66d5d8fe0a"
server_key_iv: "69d42a24d5ea2ae7e537bdc131c68c9c"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.474406 7901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "7c1188c4937a46caa9cf1d14cbcb7779"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "6720e1e03f122d81dc062d66d5d8fe0a"
server_key_iv: "69d42a24d5ea2ae7e537bdc131c68c9c"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.475921 7901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:28.477054 7916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.477406 7901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:28.477527 7901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "7c1188c4937a46caa9cf1d14cbcb7779"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "6720e1e03f122d81dc062d66d5d8fe0a"
server_key_iv: "69d42a24d5ea2ae7e537bdc131c68c9c"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.477619 7901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:28.489548 7901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:28.492300 7901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:28.492478 7901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:28.492739 7901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:28.493106 7901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:28.493165 7901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.493198 7901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:28.493229 7901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.499626 7901 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:42105
I20260923 23:05:28.499670 8029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:42105 every 8 connection(s)
I20260923 23:05:28.500031 7901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:28.500330 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 7901
I20260923 23:05:28.500447 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:28.500636 5416 external_mini_cluster.cc:1428] Setting key 4d0acbca153807abf62c074cfff2d420
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[7794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204728, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:05:28.514133 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:28.501038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:54829 (local address 127.5.74.62:33423)
0923 23:05:28.501087 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:28.501089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:28.501444 (+ 355us) server_negotiation.cc:407] Connection header received
0923 23:05:28.502049 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:28.502050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:28.502064 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:28.502149 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:28.503578 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.504140 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.505909 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.506331 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.506530 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:28.510027 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:28.510030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:28.510030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:28.510044 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:28.512289 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:28.512899 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:28.512900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:28.512900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:28.512962 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:28.513221 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:28.513223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:28.513223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:28.513371 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:28.513542 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:28.513979 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 23:05:28.514062 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:28.514564 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:28.501043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:33423 (local address 127.5.74.1:54829)
0923 23:05:28.501259 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:28.501308 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:28.501898 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:28.502325 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:28.502329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:28.502989 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:28.503446 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:28.503455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.504330 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.504331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:28.505773 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:28.505777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.506624 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.506628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:28.506903 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:28.507838 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:28.507848 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:28.509894 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:28.512532 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:28.512549 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:28.512552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:28.512806 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:28.513038 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:28.513039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:28.513039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:28.513145 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:28.513569 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:28.513575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:28.513714 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 23:05:28.513921 (+ 207us) client_negotiation.cc:240] Negotiation successful
0923 23:05:28.514073 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
I20260923 23:05:28.515448 8030 heartbeater.cc:347] Connected to a master server at 127.5.74.62:33423
I20260923 23:05:28.515578 8030 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:28.515771 8030 heartbeater.cc:511] Master 127.5.74.62:33423 requested a full tablet report, sending...
I20260923 23:05:28.516389 7838 ts_manager.cc:194] Registered new tserver with Master: 7c1188c4937a46caa9cf1d14cbcb7779 (127.5.74.1:42105)
I20260923 23:05:28.517364 7838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:54829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:28.536092 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:33423
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:28.631568 8035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:28.631803 8035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:28.631830 8035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:28.633301 8035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:28.633358 8035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:28.633392 8035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:28.635350 8035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:28.635660 8035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:28.635926 8035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:28.641584 8041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:28.641566 8040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:28.641566 8043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:28.642091 8035 server_base.cc:1034] running on GCE node
I20260923 23:05:28.642339 8035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:28.642661 8035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:28.643841 8035 hybrid_clock.cc:648] HybridClock initialized: now 1790204728643825 us; error 29 us; skew 500 ppm
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[7794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204728, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:28.646772 8035 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:28.647272 8035 webserver.cc:460] Webserver started at http://127.5.74.2:45367/ using document root <none> and password file <none>
I20260923 23:05:28.647526 8035 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:28.647609 8035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:28.647765 8035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:28.649008 8035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "a00b6089d31e46bb948a9155ec7d959d"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "a03e0573b71c2af71ba7b895403f1a8c"
server_key_iv: "b8c89e53449b4e63485de9b57f803415"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.649396 8035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "a00b6089d31e46bb948a9155ec7d959d"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "a03e0573b71c2af71ba7b895403f1a8c"
server_key_iv: "b8c89e53449b4e63485de9b57f803415"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.650957 8035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:28.652062 8050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.652529 8035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:28.652834 8035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "a00b6089d31e46bb948a9155ec7d959d"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "a03e0573b71c2af71ba7b895403f1a8c"
server_key_iv: "b8c89e53449b4e63485de9b57f803415"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.652944 8035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:28.663555 8035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:28.666468 8035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:28.666669 8035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:28.666952 8035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:28.667306 8035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:28.667377 8035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.667496 8035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:28.667532 8035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.674670 8035 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:37457
I20260923 23:05:28.674763 8163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:37457 every 8 connection(s)
I20260923 23:05:28.675122 8035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:05:28.682404 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 8035
I20260923 23:05:28.682528 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:28.682729 5416 external_mini_cluster.cc:1428] Setting key 8a142f599d3600dd318d92bf6a1530a6
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[7794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204728, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:28.687523 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:28.676309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:55789 (local address 127.5.74.62:33423)
0923 23:05:28.676363 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:28.676365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:28.676546 (+ 181us) server_negotiation.cc:407] Connection header received
0923 23:05:28.677478 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:28.677484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:28.677496 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:28.677555 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:28.678934 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.679481 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.680767 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.681048 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.681122 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:28.683876 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:28.683878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:28.683878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:28.683888 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:28.685774 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:28.686391 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:28.686393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:28.686394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:28.686465 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:28.686763 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:28.686764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:28.686764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:28.686906 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:28.686995 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:28.687309 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 23:05:28.687411 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:05:28.687852 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:28.676017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:33423 (local address 127.5.74.2:55789)
0923 23:05:28.676349 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:28.676420 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:28.677293 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:28.677618 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:28.677622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:28.678385 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:28.678747 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:28.678753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.679582 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.679583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:28.680607 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:28.680610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.681154 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.681155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:28.681275 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:28.681879 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:28.681889 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:28.683773 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:28.685891 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:28.685900 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:28.685904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:28.686270 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:28.686545 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:28.686546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:28.686546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:28.686680 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:28.686984 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:28.686991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:28.687158 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 23:05:28.687289 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 23:05:28.687493 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":72,"threads_started":1}
I20260923 23:05:28.688591 8164 heartbeater.cc:347] Connected to a master server at 127.5.74.62:33423
I20260923 23:05:28.688730 8164 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:28.688952 8164 heartbeater.cc:511] Master 127.5.74.62:33423 requested a full tablet report, sending...
I20260923 23:05:28.689410 7838 ts_manager.cc:194] Registered new tserver with Master: a00b6089d31e46bb948a9155ec7d959d (127.5.74.2:37457)
I20260923 23:05:28.689816 7838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:55789
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:28.717072 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:33423
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:28.807914 8169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:28.808143 8169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:28.808250 8169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:28.809584 8169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:28.809631 8169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:28.809661 8169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:28.811180 8169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:28.811470 8169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:28.811714 8169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:28.817731 8174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:28.817929 8169 server_base.cc:1034] running on GCE node
W20260923 23:05:28.817767 8177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:28.817729 8175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:28.818267 8169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:28.818555 8169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:28.819785 8169 hybrid_clock.cc:648] HybridClock initialized: now 1790204728819747 us; error 44 us; skew 500 ppm
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[7794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204728, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:28.822695 8169 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:28.823160 8169 webserver.cc:460] Webserver started at http://127.5.74.3:36755/ using document root <none> and password file <none>
I20260923 23:05:28.823441 8169 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:28.823522 8169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:28.823683 8169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:28.824846 8169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "b823974174544414ab3d9e904b7768f9"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "f765da58a360c7ce3637bd3713900e21"
server_key_iv: "cfa7393ba35843a52052bd0ede5f7234"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.825229 8169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "b823974174544414ab3d9e904b7768f9"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "f765da58a360c7ce3637bd3713900e21"
server_key_iv: "cfa7393ba35843a52052bd0ede5f7234"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.826685 8169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:05:28.827505 8184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.827695 8169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:28.827915 8169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "b823974174544414ab3d9e904b7768f9"
format_stamp: "Formatted at 2026-09-23 23:05:28 on dist-test-slave-j4c5"
server_key: "f765da58a360c7ce3637bd3713900e21"
server_key_iv: "cfa7393ba35843a52052bd0ede5f7234"
server_key_version: "encryptionkey@0"
I20260923 23:05:28.828024 8169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:28.842368 8169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:28.845198 8169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:28.845384 8169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:28.845670 8169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:28.846028 8169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:28.846067 8169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.846093 8169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:28.846107 8169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:28.854799 8169 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:35023
I20260923 23:05:28.854876 8297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:35023 every 8 connection(s)
I20260923 23:05:28.855244 8169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[7794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204728, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:28.863004 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 8169
I20260923 23:05:28.863113 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:28.863399 5416 external_mini_cluster.cc:1428] Setting key dd4ff072894aede41c1d971d39ba240b
I20260923 23:05:28.865851 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:28.856253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:48091 (local address 127.5.74.62:33423)
0923 23:05:28.856356 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:28.856358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:28.856583 (+ 225us) server_negotiation.cc:407] Connection header received
0923 23:05:28.857126 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:28.857127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:28.857139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:28.857193 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:28.858221 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.858727 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.859789 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.860021 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.860112 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:28.862491 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:28.862493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:28.862494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:28.862502 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:28.864410 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:28.864925 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:28.864926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:28.864926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:28.864994 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:28.865218 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:28.865219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:28.865219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:28.865345 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:28.865408 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:28.865715 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 23:05:28.865786 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 23:05:28.866117 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:28.856259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:33423 (local address 127.5.74.3:48091)
0923 23:05:28.856455 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:28.856494 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:28.857014 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:28.857281 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:28.857286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:28.857904 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:28.858107 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:28.858112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.858830 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.858831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:28.859689 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:28.859691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:28.860127 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:28.860128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:28.860296 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:28.860938 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:28.860947 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:28.862380 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:28.864515 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:28.864522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:28.864524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:28.864778 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:28.865057 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:28.865058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:28.865058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:28.865162 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:28.865453 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:28.865458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:28.865591 (+ 133us) client_negotiation.cc:769] Sending connection context
0923 23:05:28.865684 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 23:05:28.865822 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260923 23:05:28.866803 8298 heartbeater.cc:347] Connected to a master server at 127.5.74.62:33423
I20260923 23:05:28.866911 8298 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:28.867129 8298 heartbeater.cc:511] Master 127.5.74.62:33423 requested a full tablet report, sending...
I20260923 23:05:28.867651 7838 ts_manager.cc:194] Registered new tserver with Master: b823974174544414ab3d9e904b7768f9 (127.5.74.3:35023)
I20260923 23:05:28.868080 7838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:48091
I20260923 23:05:28.875293 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 23:05:28.887596 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:28.886528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.5.74.62:33423)
0923 23:05:28.886625 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:28.886628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:28.886693 (+ 65us) server_negotiation.cc:407] Connection header received
0923 23:05:28.886740 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:28.886741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:28.886751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:28.886865 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:28.887492 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34116: BlockingRecv error: recv got EOF from 127.0.0.1:34116 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 23:05:28.888461 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 7901
I20260923 23:05:28.893234 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 8035
I20260923 23:05:28.897650 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 8169
I20260923 23:05:28.902323 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 7807
2026-09-23T23:05:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2045 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[8310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[8310](info): set up 2 sockets
Sep 23 23:05:28 dist-test-slave-j4c5 krb5kdc[8310](info): commencing operation
krb5kdc: starting...
W20260923 23:05:29.993067 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:30 dist-test-slave-j4c5 krb5kdc[8310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:30.133159 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34825
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:34825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:30.223893 8323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:30.224133 8323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:30.224332 8323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:30.225605 8323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:30.225651 8323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:30.225664 8323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:30.225678 8323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:30.225692 8323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:30.227258 8323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:34825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:30.227494 8323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:30.227686 8323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:30.233444 8331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:30.233397 8328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:30.233417 8329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:30.233819 8323 server_base.cc:1034] running on GCE node
I20260923 23:05:30.234126 8323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:30.234447 8323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:30.235652 8323 hybrid_clock.cc:648] HybridClock initialized: now 1790204730235639 us; error 30 us; skew 500 ppm
Sep 23 23:05:30 dist-test-slave-j4c5 krb5kdc[8310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204730, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:30.246412 8323 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:30.246944 8323 webserver.cc:460] Webserver started at http://127.5.74.62:45285/ using document root <none> and password file <none>
I20260923 23:05:30.247192 8323 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:30.247267 8323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:30.247431 8323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:30.248715 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "fc41c388c5774ce2b39ea979360e5daf"
format_stamp: "Formatted at 2026-09-23 23:05:30 on dist-test-slave-j4c5"
server_key: "b6b5328b5d848349d3abf20773d0faf4"
server_key_iv: "f5bf8fcdea97ccad531b694174ffec8c"
server_key_version: "encryptionkey@0"
I20260923 23:05:30.249110 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "fc41c388c5774ce2b39ea979360e5daf"
format_stamp: "Formatted at 2026-09-23 23:05:30 on dist-test-slave-j4c5"
server_key: "b6b5328b5d848349d3abf20773d0faf4"
server_key_iv: "f5bf8fcdea97ccad531b694174ffec8c"
server_key_version: "encryptionkey@0"
I20260923 23:05:30.250663 8323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:30.251657 8338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:30.251971 8323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:30.252135 8323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "fc41c388c5774ce2b39ea979360e5daf"
format_stamp: "Formatted at 2026-09-23 23:05:30 on dist-test-slave-j4c5"
server_key: "b6b5328b5d848349d3abf20773d0faf4"
server_key_iv: "f5bf8fcdea97ccad531b694174ffec8c"
server_key_version: "encryptionkey@0"
I20260923 23:05:30.252291 8323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:30.306636 8323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:30.310056 8323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:30.310233 8323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:30.314710 8323 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:34825
I20260923 23:05:30.314761 8390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:34825 every 8 connection(s)
I20260923 23:05:30.315114 8323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:30.316095 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:30.318692 8391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf: Bootstrap starting.
I20260923 23:05:30.319269 8391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:30.319535 8391 log.cc:824] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:30.320125 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 8323
I20260923 23:05:30.320250 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:30.320379 8391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf: No bootstrap required, opened a new log
I20260923 23:05:30.320513 5416 external_mini_cluster.cc:1428] Setting key 9c9f18a177aea963f981d82d59fad0de
I20260923 23:05:30.322932 8391 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc41c388c5774ce2b39ea979360e5daf" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34825 } }
I20260923 23:05:30.323081 8391 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:30.323101 8391 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc41c388c5774ce2b39ea979360e5daf, State: Initialized, Role: FOLLOWER
I20260923 23:05:30.323261 8391 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc41c388c5774ce2b39ea979360e5daf" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34825 } }
I20260923 23:05:30.323374 8391 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:30.323419 8391 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:30.323463 8391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:30.324040 8391 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc41c388c5774ce2b39ea979360e5daf" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34825 } }
I20260923 23:05:30.324165 8391 leader_election.cc:302] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc41c388c5774ce2b39ea979360e5daf; no voters:
I20260923 23:05:30.324554 8391 leader_election.cc:288] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:30.324616 8396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:30.324823 8396 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [term 1 LEADER]: Becoming Leader. State: Replica: fc41c388c5774ce2b39ea979360e5daf, State: Running, Role: LEADER
I20260923 23:05:30.324836 8391 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:30.324954 8396 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc41c388c5774ce2b39ea979360e5daf" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34825 } }
I20260923 23:05:30.325522 8397 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc41c388c5774ce2b39ea979360e5daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc41c388c5774ce2b39ea979360e5daf" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34825 } } }
I20260923 23:05:30.325845 8397 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:30.325512 8396 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc41c388c5774ce2b39ea979360e5daf. Latest consensus state: current_term: 1 leader_uuid: "fc41c388c5774ce2b39ea979360e5daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc41c388c5774ce2b39ea979360e5daf" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34825 } } }
I20260923 23:05:30.326570 8396 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:30.326792 8405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:30.327339 8405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:30.329424 8405 catalog_manager.cc:1347] Generated new cluster ID: ea8f7ef1265c4846b70e8fd5f9564572
I20260923 23:05:30.329488 8405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:05:30 dist-test-slave-j4c5 krb5kdc[8310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:30.335484 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:30.321009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.5.74.62:34825)
0923 23:05:30.321179 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:30.321186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:30.321211 (+ 25us) server_negotiation.cc:407] Connection header received
0923 23:05:30.322935 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:30.322944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:30.323006 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:30.323219 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:30.324026 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:30.325819 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:30.326904 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:30.327557 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:30.327671 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:30.330660 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:30.330665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:30.330666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:30.330678 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:30.333375 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:30.334071 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:30.334077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:30.334077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:30.334249 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:30.334474 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:30.334474 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:30.334475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:30.334844 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:30.335021 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:30.335105 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 23:05:30.335262 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260923 23:05:30.336475 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:05:30.339494 8405 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:30.340364 8405 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:30.356251 8405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc41c388c5774ce2b39ea979360e5daf: Generated new TSK 0
I20260923 23:05:30.356518 8405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:05:30.360857 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:30.356593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.5.74.62:34825)
0923 23:05:30.356644 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:30.356646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:30.356801 (+ 155us) server_negotiation.cc:407] Connection header received
0923 23:05:30.356891 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:30.356892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:30.356912 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:30.356958 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:30.357446 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:30.358039 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:30.359143 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:30.359389 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:30.359456 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:30.360210 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:30.360212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:30.360212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:30.360460 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:30.360489 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:30.360496 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:30.360546 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:30.360658 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 23:05:30.360684 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:05:30.363413 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 8323
2026-09-23T23:05:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1458 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:30 dist-test-slave-j4c5 krb5kdc[8424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:30 dist-test-slave-j4c5 krb5kdc[8424](info): set up 2 sockets
Sep 23 23:05:30 dist-test-slave-j4c5 krb5kdc[8424](info): commencing operation
krb5kdc: starting...
W20260923 23:05:31.440531 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:31 dist-test-slave-j4c5 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:31.573421 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:43475
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:31.666524 8437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:31.666785 8437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:31.666811 8437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:31.668195 8437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:31.668275 8437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:31.668298 8437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:31.668320 8437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:31.668354 8437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:31.670329 8437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:43475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:31.670699 8437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:31.670979 8437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:31.676733 8442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:31.676699 8445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:31.676725 8443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:31.676744 8437 server_base.cc:1034] running on GCE node
I20260923 23:05:31.677227 8437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:31.677592 8437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:31.678769 8437 hybrid_clock.cc:648] HybridClock initialized: now 1790204731678728 us; error 88 us; skew 500 ppm
Sep 23 23:05:31 dist-test-slave-j4c5 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204731, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:31.682232 8437 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:31.683254 8437 webserver.cc:460] Webserver started at http://127.5.74.62:41979/ using document root <none> and password file <none>
I20260923 23:05:31.683588 8437 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:31.683641 8437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:31.683758 8437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:31.684924 8437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "2d3dd079811f40259aefaf4f7f837f53"
format_stamp: "Formatted at 2026-09-23 23:05:31 on dist-test-slave-j4c5"
server_key: "763ba4847b145c4b2dcdf0bf104c368e"
server_key_iv: "1bbb1c1a28d16fc5534d3a84c8a8f7df"
server_key_version: "encryptionkey@0"
I20260923 23:05:31.685247 8437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "2d3dd079811f40259aefaf4f7f837f53"
format_stamp: "Formatted at 2026-09-23 23:05:31 on dist-test-slave-j4c5"
server_key: "763ba4847b145c4b2dcdf0bf104c368e"
server_key_iv: "1bbb1c1a28d16fc5534d3a84c8a8f7df"
server_key_version: "encryptionkey@0"
I20260923 23:05:31.686626 8437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:31.687471 8452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:31.687700 8437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:31.687811 8437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "2d3dd079811f40259aefaf4f7f837f53"
format_stamp: "Formatted at 2026-09-23 23:05:31 on dist-test-slave-j4c5"
server_key: "763ba4847b145c4b2dcdf0bf104c368e"
server_key_iv: "1bbb1c1a28d16fc5534d3a84c8a8f7df"
server_key_version: "encryptionkey@0"
I20260923 23:05:31.687892 8437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:31.705305 8437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:31.707679 8437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:31.707863 8437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:31.712327 8437 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:43475
I20260923 23:05:31.712404 8504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:43475 every 8 connection(s)
I20260923 23:05:31.712766 8437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:31.713719 8505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:31.715952 8505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53: Bootstrap starting.
I20260923 23:05:31.716602 8505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:31.716842 8505 log.cc:824] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:31.717506 8505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53: No bootstrap required, opened a new log
I20260923 23:05:31.719198 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 8437
I20260923 23:05:31.719146 8505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3dd079811f40259aefaf4f7f837f53" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43475 } }
I20260923 23:05:31.719324 8505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:31.719380 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:31.719349 8505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3dd079811f40259aefaf4f7f837f53, State: Initialized, Role: FOLLOWER
I20260923 23:05:31.719552 8505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3dd079811f40259aefaf4f7f837f53" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43475 } }
I20260923 23:05:31.719611 5416 external_mini_cluster.cc:1428] Setting key 5c118eae513e766107e7da953a661ca4
I20260923 23:05:31.719642 8505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:31.719676 8505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:31.719735 8505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:31.720463 8505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3dd079811f40259aefaf4f7f837f53" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43475 } }
I20260923 23:05:31.720590 8505 leader_election.cc:302] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d3dd079811f40259aefaf4f7f837f53; no voters:
I20260923 23:05:31.720772 8505 leader_election.cc:288] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:31.720842 8510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:31.721017 8510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [term 1 LEADER]: Becoming Leader. State: Replica: 2d3dd079811f40259aefaf4f7f837f53, State: Running, Role: LEADER
I20260923 23:05:31.721037 8505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:31.721114 8510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3dd079811f40259aefaf4f7f837f53" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43475 } }
I20260923 23:05:31.721504 8510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d3dd079811f40259aefaf4f7f837f53. Latest consensus state: current_term: 1 leader_uuid: "2d3dd079811f40259aefaf4f7f837f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3dd079811f40259aefaf4f7f837f53" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43475 } } }
I20260923 23:05:31.721508 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d3dd079811f40259aefaf4f7f837f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3dd079811f40259aefaf4f7f837f53" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43475 } } }
I20260923 23:05:31.721681 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:31.721658 8510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:31.721990 8518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:31.722563 8518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:31.724136 8518 catalog_manager.cc:1347] Generated new cluster ID: fa57d32542144e709228b6415f863179
I20260923 23:05:31.724215 8518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:05:31 dist-test-slave-j4c5 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:31.732468 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:31.720185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.5.74.62:43475)
0923 23:05:31.720786 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:31.720796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:31.720834 (+ 38us) server_negotiation.cc:407] Connection header received
0923 23:05:31.720964 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:31.720970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:31.721028 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:31.722211 (+ 1183us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:31.722350 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:31.723780 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:31.724919 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:31.725637 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:31.725763 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:31.728640 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:31.728644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:31.728644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:31.728656 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:31.730725 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:31.731153 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:31.731156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:31.731156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:31.731284 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:31.731510 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:31.731510 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:31.731511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:31.731820 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:31.731955 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:31.732077 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 23:05:31.732240 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":514,"threads_started":1}
I20260923 23:05:31.739872 8518 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:31.740552 8518 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:31.743752 8518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d3dd079811f40259aefaf4f7f837f53: Generated new TSK 0
I20260923 23:05:31.743973 8518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:31.779263 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43475
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:31.868044 8531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:31.868306 8531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:31.868333 8531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:31.869541 8531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:31.869582 8531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:31.869609 8531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:31.871371 8531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:31.871608 8531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:31.871802 8531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:31.877454 8537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:31.877467 8539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:31.877472 8536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:31.877617 8531 server_base.cc:1034] running on GCE node
I20260923 23:05:31.878093 8531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:31.878368 8531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:31.879530 8531 hybrid_clock.cc:648] HybridClock initialized: now 1790204731879522 us; error 37 us; skew 500 ppm
Sep 23 23:05:31 dist-test-slave-j4c5 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204731, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:31.882284 8531 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:31.882714 8531 webserver.cc:460] Webserver started at http://127.5.74.1:43151/ using document root <none> and password file <none>
I20260923 23:05:31.882961 8531 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:31.883031 8531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:31.883183 8531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:31.884395 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "2208f01961f84139a3a7f30f461bbb27"
format_stamp: "Formatted at 2026-09-23 23:05:31 on dist-test-slave-j4c5"
server_key: "5e3a54f26301638cdb84da6b570f6104"
server_key_iv: "d073b742bc9f19c80e68b9f32b801a3f"
server_key_version: "encryptionkey@0"
I20260923 23:05:31.884768 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "2208f01961f84139a3a7f30f461bbb27"
format_stamp: "Formatted at 2026-09-23 23:05:31 on dist-test-slave-j4c5"
server_key: "5e3a54f26301638cdb84da6b570f6104"
server_key_iv: "d073b742bc9f19c80e68b9f32b801a3f"
server_key_version: "encryptionkey@0"
I20260923 23:05:31.886219 8531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:31.887069 8546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:31.887269 8531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:31.887389 8531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "2208f01961f84139a3a7f30f461bbb27"
format_stamp: "Formatted at 2026-09-23 23:05:31 on dist-test-slave-j4c5"
server_key: "5e3a54f26301638cdb84da6b570f6104"
server_key_iv: "d073b742bc9f19c80e68b9f32b801a3f"
server_key_version: "encryptionkey@0"
I20260923 23:05:31.887470 8531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:31.909788 8531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:31.912273 8531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:31.912446 8531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:31.912694 8531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:31.913021 8531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:31.913077 8531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:31.913120 8531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:31.913144 8531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:31.918849 8531 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:37567
I20260923 23:05:31.918895 8659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:37567 every 8 connection(s)
I20260923 23:05:31.919256 8531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:31.925067 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 8531
I20260923 23:05:31.925156 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:31.925429 5416 external_mini_cluster.cc:1428] Setting key 74107ed8492b49a6f1aef0417d254b2e
Sep 23 23:05:31 dist-test-slave-j4c5 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204731, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
I20260923 23:05:31.937801 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:31.920075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:57365 (local address 127.5.74.62:43475)
0923 23:05:31.920143 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:31.920183 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:31.920324 (+ 141us) server_negotiation.cc:407] Connection header received
0923 23:05:31.920781 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:31.920782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:31.920794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:31.920898 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:31.921978 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:31.922569 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:31.923808 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:31.924013 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:31.924073 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:31.934054 (+ 9981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:31.934056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:31.934056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:31.934074 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:31.935841 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:31.936417 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:31.936419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:31.936419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:31.936486 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:31.936963 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:31.936964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:31.936964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:31.937136 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:31.937210 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:31.937617 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 23:05:31.937707 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:05:31.938079 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:31.920011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:43475 (local address 127.5.74.1:57365)
0923 23:05:31.920185 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:31.920228 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:31.920698 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:31.920954 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:31.920957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:31.921616 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:31.921825 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:31.921830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:31.922678 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:31.922679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:31.923706 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:31.923709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:31.924097 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:31.924098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:31.924225 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:31.931599 (+ 7374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:31.931615 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:31.933906 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:31.935961 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:31.935972 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:31.935984 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:31.936335 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:31.936566 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:31.936596 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:31.936596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:31.936867 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:31.937242 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:31.937250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:31.937455 (+ 205us) client_negotiation.cc:769] Sending connection context
0923 23:05:31.937580 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 23:05:31.937815 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260923 23:05:31.938791 8660 heartbeater.cc:347] Connected to a master server at 127.5.74.62:43475
I20260923 23:05:31.938910 8660 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:31.939117 8660 heartbeater.cc:511] Master 127.5.74.62:43475 requested a full tablet report, sending...
I20260923 23:05:31.939658 8469 ts_manager.cc:194] Registered new tserver with Master: 2208f01961f84139a3a7f30f461bbb27 (127.5.74.1:37567)
I20260923 23:05:31.940371 8469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:57365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:31.956105 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43475
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:32.042734 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:32.042941 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:32.042965 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:32.044185 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:32.044226 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:32.044253 8665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:32.045689 8665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:32.045926 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:32.046183 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:05:32.051436 8665 server_base.cc:1034] running on GCE node
W20260923 23:05:32.051406 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:32.051582 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:32.051411 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:32.051831 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:32.052094 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:32.053282 8665 hybrid_clock.cc:648] HybridClock initialized: now 1790204732053267 us; error 28 us; skew 500 ppm
Sep 23 23:05:32 dist-test-slave-j4c5 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204732, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:32.055785 8665 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:32.056310 8665 webserver.cc:460] Webserver started at http://127.5.74.2:35529/ using document root <none> and password file <none>
I20260923 23:05:32.056536 8665 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:32.056607 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:32.056764 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:32.057883 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "6f279f13819b434180b08dc3296d602d"
format_stamp: "Formatted at 2026-09-23 23:05:32 on dist-test-slave-j4c5"
server_key: "b36849f216844ffad3ade3c67e56b485"
server_key_iv: "c150fb3f5da1ac31ad2870e7f686af12"
server_key_version: "encryptionkey@0"
I20260923 23:05:32.058260 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "6f279f13819b434180b08dc3296d602d"
format_stamp: "Formatted at 2026-09-23 23:05:32 on dist-test-slave-j4c5"
server_key: "b36849f216844ffad3ade3c67e56b485"
server_key_iv: "c150fb3f5da1ac31ad2870e7f686af12"
server_key_version: "encryptionkey@0"
I20260923 23:05:32.059711 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:32.060684 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:32.060946 8665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:05:32.061057 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "6f279f13819b434180b08dc3296d602d"
format_stamp: "Formatted at 2026-09-23 23:05:32 on dist-test-slave-j4c5"
server_key: "b36849f216844ffad3ade3c67e56b485"
server_key_iv: "c150fb3f5da1ac31ad2870e7f686af12"
server_key_version: "encryptionkey@0"
I20260923 23:05:32.061157 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:32.080824 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:32.083488 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:32.083719 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:32.084029 8665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:32.084409 8665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:32.084445 8665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:32.084468 8665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:32.084517 8665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:32.092044 8665 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:43203
I20260923 23:05:32.092118 8793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:43203 every 8 connection(s)
I20260923 23:05:32.092463 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:05:32 dist-test-slave-j4c5 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204732, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:32.102247 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 8665
I20260923 23:05:32.102382 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:32.102602 5416 external_mini_cluster.cc:1428] Setting key 994263d83cae65d0f987c9ec547c9eaf
I20260923 23:05:32.105448 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:32.093451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:47441 (local address 127.5.74.62:43475)
0923 23:05:32.093552 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:32.093554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:32.093707 (+ 153us) server_negotiation.cc:407] Connection header received
0923 23:05:32.094218 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:32.094218 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:32.094229 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:32.094289 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:32.095452 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.095985 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.097358 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.097554 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.097625 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:32.100933 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:32.100936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:32.100936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:32.100945 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:32.102929 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:32.103699 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:32.103700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:32.103701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:32.103774 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:32.104524 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:32.104526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:32.104526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:32.104674 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:32.104781 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:32.105169 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 23:05:32.105311 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 23:05:32.106138 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:32.093365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:43475 (local address 127.5.74.2:47441)
0923 23:05:32.093539 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:32.093589 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:32.094108 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:32.094380 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:32.094384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:32.094995 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:32.095222 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:32.095227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.096099 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.096100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:32.097213 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:32.097217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.097666 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.097668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:32.097792 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:32.098929 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:32.098939 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:32.100799 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:32.103063 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:32.103074 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:32.103077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:32.103569 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:32.103958 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:32.103959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:32.103959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:32.104109 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:32.104772 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:32.104779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:32.104974 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 23:05:32.105611 (+ 637us) client_negotiation.cc:240] Negotiation successful
0923 23:05:32.105844 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260923 23:05:32.106758 8794 heartbeater.cc:347] Connected to a master server at 127.5.74.62:43475
I20260923 23:05:32.106905 8794 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:32.107120 8794 heartbeater.cc:511] Master 127.5.74.62:43475 requested a full tablet report, sending...
I20260923 23:05:32.107630 8469 ts_manager.cc:194] Registered new tserver with Master: 6f279f13819b434180b08dc3296d602d (127.5.74.2:43203)
I20260923 23:05:32.108254 8469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:47441
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:32.135424 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43475
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:32.223218 8799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:32.223495 8799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:32.223538 8799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:32.225149 8799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:32.225231 8799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:32.225281 8799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:32.226887 8799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:32.227169 8799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:32.227418 8799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:32.233700 8807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:32.233709 8804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:32.233855 8799 server_base.cc:1034] running on GCE node
W20260923 23:05:32.233700 8805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:32.234362 8799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:32.234658 8799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:32.235831 8799 hybrid_clock.cc:648] HybridClock initialized: now 1790204732235771 us; error 78 us; skew 500 ppm
Sep 23 23:05:32 dist-test-slave-j4c5 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204732, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:32.238656 8799 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:32.239398 8799 webserver.cc:460] Webserver started at http://127.5.74.3:43437/ using document root <none> and password file <none>
I20260923 23:05:32.239713 8799 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:32.239841 8799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:32.240120 8799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:32.241294 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "a243bc8bfcf1471bbd5ab1ac29d38347"
format_stamp: "Formatted at 2026-09-23 23:05:32 on dist-test-slave-j4c5"
server_key: "a5f789547c3905843f7547c748d0f336"
server_key_iv: "d5052fcc699e42a1c6e6dee859507ba6"
server_key_version: "encryptionkey@0"
I20260923 23:05:32.241674 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "a243bc8bfcf1471bbd5ab1ac29d38347"
format_stamp: "Formatted at 2026-09-23 23:05:32 on dist-test-slave-j4c5"
server_key: "a5f789547c3905843f7547c748d0f336"
server_key_iv: "d5052fcc699e42a1c6e6dee859507ba6"
server_key_version: "encryptionkey@0"
I20260923 23:05:32.243052 8799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:05:32.243858 8814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:32.244050 8799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:32.244129 8799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "a243bc8bfcf1471bbd5ab1ac29d38347"
format_stamp: "Formatted at 2026-09-23 23:05:32 on dist-test-slave-j4c5"
server_key: "a5f789547c3905843f7547c748d0f336"
server_key_iv: "d5052fcc699e42a1c6e6dee859507ba6"
server_key_version: "encryptionkey@0"
I20260923 23:05:32.244243 8799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:32.258150 8799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:32.260859 8799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:32.261055 8799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:32.261359 8799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:32.261693 8799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:32.261749 8799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:32.261785 8799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:32.261832 8799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:32.268139 8799 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:43491
I20260923 23:05:32.268289 8927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:43491 every 8 connection(s)
I20260923 23:05:32.268636 8799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:05:32.271158 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 8799
I20260923 23:05:32.271260 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:32.271445 5416 external_mini_cluster.cc:1428] Setting key 8fdda37e56132fae155f6ded62fad91c
Sep 23 23:05:32 dist-test-slave-j4c5 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204732, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:32.279788 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:32.269447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:49383 (local address 127.5.74.62:43475)
0923 23:05:32.269490 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:32.269493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:32.269732 (+ 239us) server_negotiation.cc:407] Connection header received
0923 23:05:32.270226 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:32.270227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:32.270243 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:32.270333 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:32.271522 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.272286 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.273436 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.273641 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.273717 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:32.276107 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:32.276109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:32.276109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:32.276117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:32.277803 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:32.278528 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:32.278530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:32.278530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:32.278605 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:32.278950 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:32.278951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:32.278952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:32.279095 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:32.279188 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:32.279581 (+ 393us) server_negotiation.cc:299] Negotiation successful
0923 23:05:32.279683 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:05:32.280009 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:32.269412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:43475 (local address 127.5.74.3:49383)
0923 23:05:32.269582 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:32.269625 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:32.270104 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:32.270430 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:32.270434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:32.271074 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:32.271395 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:32.271401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.272401 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.272402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:32.273329 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:32.273332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.273725 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.273726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:32.273863 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:32.274433 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:32.274441 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:32.275968 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:32.277921 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:32.277929 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:32.277932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:32.278435 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:32.278684 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:32.278686 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:32.278686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:32.278865 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:32.279174 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:32.279182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:32.279358 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 23:05:32.279542 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 23:05:32.279735 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260923 23:05:32.280650 8928 heartbeater.cc:347] Connected to a master server at 127.5.74.62:43475
I20260923 23:05:32.280755 8928 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:32.280993 8928 heartbeater.cc:511] Master 127.5.74.62:43475 requested a full tablet report, sending...
I20260923 23:05:32.281567 8469 ts_manager.cc:194] Registered new tserver with Master: a243bc8bfcf1471bbd5ab1ac29d38347 (127.5.74.3:43491)
I20260923 23:05:32.281975 8469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:49383
I20260923 23:05:32.283733 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 23:05:32 dist-test-slave-j4c5 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204732, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 23:05:32 dist-test-slave-j4c5 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204732, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:32.309487 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:32.301377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.5.74.62:43475)
0923 23:05:32.301457 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:32.301459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:32.301517 (+ 58us) server_negotiation.cc:407] Connection header received
0923 23:05:32.301564 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:32.301565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:32.301574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:32.301639 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:32.302492 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.303013 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.303848 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:32.304042 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:32.304150 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:32.306608 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:32.306611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:32.306611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:32.306620 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:32.308013 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:32.308463 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:32.308464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:32.308465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:32.308529 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:32.308729 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:32.308730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:32.308731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:32.308986 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:32.309118 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:32.309225 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 23:05:32.309379 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 23:05:32.309768 8469 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40108
I20260923 23:05:32.310369 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 8531
I20260923 23:05:32.315673 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 8665
I20260923 23:05:32.320030 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 8799
I20260923 23:05:32.325088 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 8437
2026-09-23T23:05:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1964 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:32 dist-test-slave-j4c5 krb5kdc[8940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:32 dist-test-slave-j4c5 krb5kdc[8940](info): set up 2 sockets
Sep 23 23:05:32 dist-test-slave-j4c5 krb5kdc[8940](info): commencing operation
krb5kdc: starting...
W20260923 23:05:33.396611 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:33 dist-test-slave-j4c5 krb5kdc[8940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:33.529105 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:40635
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:33.617899 8953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:33.618187 8953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:33.618238 8953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:33.619493 8953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:33.619575 8953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:33.619602 8953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:33.619627 8953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:33.619668 8953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:33.621404 8953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:40635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:33.621721 8953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:33.621978 8953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:33.627425 8958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:33.627489 8959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:33.627779 8953 server_base.cc:1034] running on GCE node
W20260923 23:05:33.627489 8961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:33.628237 8953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:33.628648 8953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:33.629850 8953 hybrid_clock.cc:648] HybridClock initialized: now 1790204733629833 us; error 30 us; skew 500 ppm
Sep 23 23:05:33 dist-test-slave-j4c5 krb5kdc[8940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204733, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:33.632611 8953 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:33.633049 8953 webserver.cc:460] Webserver started at http://127.5.74.62:38231/ using document root <none> and password file <none>
I20260923 23:05:33.633236 8953 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:33.633280 8953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:33.633373 8953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:33.634706 8953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "8553942c23c7439995d16f1698adee9e"
format_stamp: "Formatted at 2026-09-23 23:05:33 on dist-test-slave-j4c5"
server_key: "d43118ed67087788cbf9e1e71a9a1cbb"
server_key_iv: "722469d246580b5667bb9cc1cfdf60cf"
server_key_version: "encryptionkey@0"
I20260923 23:05:33.635063 8953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "8553942c23c7439995d16f1698adee9e"
format_stamp: "Formatted at 2026-09-23 23:05:33 on dist-test-slave-j4c5"
server_key: "d43118ed67087788cbf9e1e71a9a1cbb"
server_key_iv: "722469d246580b5667bb9cc1cfdf60cf"
server_key_version: "encryptionkey@0"
I20260923 23:05:33.636651 8953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:05:33.637571 8968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:33.637814 8953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:33.637905 8953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "8553942c23c7439995d16f1698adee9e"
format_stamp: "Formatted at 2026-09-23 23:05:33 on dist-test-slave-j4c5"
server_key: "d43118ed67087788cbf9e1e71a9a1cbb"
server_key_iv: "722469d246580b5667bb9cc1cfdf60cf"
server_key_version: "encryptionkey@0"
I20260923 23:05:33.637965 8953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:33.663033 8953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:33.665393 8953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:33.665524 8953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:33.669783 8953 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:40635
I20260923 23:05:33.669828 9020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:40635 every 8 connection(s)
I20260923 23:05:33.670440 8953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:33.671226 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:33.673605 9021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e: Bootstrap starting.
I20260923 23:05:33.674232 9021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:33.674592 9021 log.cc:824] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:33.674691 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 8953
I20260923 23:05:33.674791 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:33.674988 5416 external_mini_cluster.cc:1428] Setting key fe1b32c74d225da2e1d3cbcd30b03691
I20260923 23:05:33.675513 9021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e: No bootstrap required, opened a new log
I20260923 23:05:33.678830 9021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8553942c23c7439995d16f1698adee9e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40635 } }
I20260923 23:05:33.679070 9021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:33.679147 9021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8553942c23c7439995d16f1698adee9e, State: Initialized, Role: FOLLOWER
I20260923 23:05:33.679322 9021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8553942c23c7439995d16f1698adee9e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40635 } }
I20260923 23:05:33.679419 9021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:33.679467 9021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:33.679539 9021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:33.680347 9021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8553942c23c7439995d16f1698adee9e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40635 } }
I20260923 23:05:33.680474 9021 leader_election.cc:302] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8553942c23c7439995d16f1698adee9e; no voters:
I20260923 23:05:33.680691 9021 leader_election.cc:288] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:33.680747 9026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:33.680962 9026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [term 1 LEADER]: Becoming Leader. State: Replica: 8553942c23c7439995d16f1698adee9e, State: Running, Role: LEADER
I20260923 23:05:33.680989 9021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:33.681090 9026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8553942c23c7439995d16f1698adee9e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40635 } }
I20260923 23:05:33.681468 9026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8553942c23c7439995d16f1698adee9e. Latest consensus state: current_term: 1 leader_uuid: "8553942c23c7439995d16f1698adee9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8553942c23c7439995d16f1698adee9e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40635 } } }
I20260923 23:05:33.681489 9027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8553942c23c7439995d16f1698adee9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8553942c23c7439995d16f1698adee9e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40635 } } }
I20260923 23:05:33.681643 9027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:33.681855 9026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:33.682024 9035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:33.682790 9035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:33.684541 9035 catalog_manager.cc:1347] Generated new cluster ID: 6952cf9b9d6e4b64a3cad2c4b6729e4c
I20260923 23:05:33.684599 9035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:05:33 dist-test-slave-j4c5 krb5kdc[8940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:33.689605 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:33.675577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.5.74.62:40635)
0923 23:05:33.675788 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:33.675799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:33.675839 (+ 40us) server_negotiation.cc:407] Connection header received
0923 23:05:33.678813 (+ 2974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:33.678825 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:33.678902 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:33.679116 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:33.679947 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:33.681418 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:33.682567 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:33.683160 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:33.683293 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:33.685983 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:33.685987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:33.685988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:33.686001 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:33.687987 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:33.688460 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:33.688463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:33.688463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:33.688556 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:33.688771 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:33.688772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:33.688773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:33.689023 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:33.689210 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:33.689277 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 23:05:33.689445 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260923 23:05:33.720890 9035 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:33.721665 9035 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:33.731947 9035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8553942c23c7439995d16f1698adee9e: Generated new TSK 0
I20260923 23:05:33.732204 9035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:33.775199 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40635
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:33.862149 9047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:33.862409 9047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:33.862447 9047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:33.863632 9047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:33.863705 9047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:33.863747 9047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:33.865329 9047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:33.865622 9047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:33.865866 9047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:33.871054 9052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:33.871078 9047 server_base.cc:1034] running on GCE node
W20260923 23:05:33.871038 9053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:33.871043 9055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:33.871579 9047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:33.871869 9047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:33.873049 9047 hybrid_clock.cc:648] HybridClock initialized: now 1790204733873002 us; error 59 us; skew 500 ppm
Sep 23 23:05:33 dist-test-slave-j4c5 krb5kdc[8940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204733, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:33.875944 9047 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:33.876489 9047 webserver.cc:460] Webserver started at http://127.5.74.1:40813/ using document root <none> and password file <none>
I20260923 23:05:33.876729 9047 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:33.876804 9047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:33.876952 9047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:33.878043 9047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "cae7fef3a9854839ac154b199ef4fb9a"
format_stamp: "Formatted at 2026-09-23 23:05:33 on dist-test-slave-j4c5"
server_key: "dcc0d85324d84275205f1e5cbc7bfefa"
server_key_iv: "c1a197337c1fc22bd4223a53031250b6"
server_key_version: "encryptionkey@0"
I20260923 23:05:33.878415 9047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "cae7fef3a9854839ac154b199ef4fb9a"
format_stamp: "Formatted at 2026-09-23 23:05:33 on dist-test-slave-j4c5"
server_key: "dcc0d85324d84275205f1e5cbc7bfefa"
server_key_iv: "c1a197337c1fc22bd4223a53031250b6"
server_key_version: "encryptionkey@0"
I20260923 23:05:33.879834 9047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:05:33.880699 9062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:33.880875 9047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:05:33.880981 9047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "cae7fef3a9854839ac154b199ef4fb9a"
format_stamp: "Formatted at 2026-09-23 23:05:33 on dist-test-slave-j4c5"
server_key: "dcc0d85324d84275205f1e5cbc7bfefa"
server_key_iv: "c1a197337c1fc22bd4223a53031250b6"
server_key_version: "encryptionkey@0"
I20260923 23:05:33.881059 9047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:33.900643 9047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:33.902876 9047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:33.903151 9047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:33.903406 9047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:33.903731 9047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:33.903789 9047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:33.903825 9047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:33.903867 9047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:33.910122 9047 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:36819
I20260923 23:05:33.910171 9175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:36819 every 8 connection(s)
I20260923 23:05:33.910558 9047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:33.911183 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 9047
I20260923 23:05:33.911319 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:33.911561 5416 external_mini_cluster.cc:1428] Setting key f6eaf2790ef2685f0a7534769651d4d0
Sep 23 23:05:33 dist-test-slave-j4c5 krb5kdc[8940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204733, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:05:33.922646 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:33.911422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:48659 (local address 127.5.74.62:40635)
0923 23:05:33.911481 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:33.911483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:33.911692 (+ 209us) server_negotiation.cc:407] Connection header received
0923 23:05:33.913077 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:33.913078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:33.913092 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:33.913152 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:33.914130 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:33.914975 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:33.916214 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:33.916451 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:33.916558 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:33.919190 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:33.919192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:33.919192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:33.919204 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:33.920992 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:33.921536 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:33.921538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:33.921538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:33.921604 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:33.921863 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:33.921865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:33.921865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:33.921996 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:33.922080 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:33.922448 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 23:05:33.922547 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:05:33.922868 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:33.911373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:40635 (local address 127.5.74.1:48659)
0923 23:05:33.911535 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:33.911586 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:33.912937 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:33.913259 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:33.913263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:33.913690 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:33.913992 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:33.913998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:33.915080 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:33.915081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:33.916076 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:33.916080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:33.916583 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:33.916584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:33.916726 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:33.917337 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:33.917347 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:33.919066 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:33.921129 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:33.921137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:33.921140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:33.921438 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:33.921676 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:33.921677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:33.921677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:33.921791 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:33.922101 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:33.922107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:33.922292 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 23:05:33.922420 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 23:05:33.922594 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
I20260923 23:05:33.923337 9176 heartbeater.cc:347] Connected to a master server at 127.5.74.62:40635
I20260923 23:05:33.923465 9176 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:33.923691 9176 heartbeater.cc:511] Master 127.5.74.62:40635 requested a full tablet report, sending...
I20260923 23:05:33.924427 8985 ts_manager.cc:194] Registered new tserver with Master: cae7fef3a9854839ac154b199ef4fb9a (127.5.74.1:36819)
I20260923 23:05:33.925441 8985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:48659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:33.941658 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40635
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:34.027386 9181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:34.027614 9181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:34.027642 9181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:34.028903 9181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:34.028955 9181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:34.028985 9181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:34.030553 9181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:34.030836 9181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:34.031098 9181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:34.036578 9189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:34.036578 9186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:34.036578 9187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:34.036655 9181 server_base.cc:1034] running on GCE node
I20260923 23:05:34.037021 9181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:34.037304 9181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:34.038491 9181 hybrid_clock.cc:648] HybridClock initialized: now 1790204734038486 us; error 55 us; skew 500 ppm
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[8940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:34.041462 9181 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:34.041968 9181 webserver.cc:460] Webserver started at http://127.5.74.2:37269/ using document root <none> and password file <none>
I20260923 23:05:34.042158 9181 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:34.042196 9181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:34.042289 9181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:34.043323 9181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "2d1c1d324d2249daad78dec390a7fbd6"
format_stamp: "Formatted at 2026-09-23 23:05:34 on dist-test-slave-j4c5"
server_key: "88e655613a841b5f3e986246a85d3573"
server_key_iv: "cf0352c3eb950fac9e30c486b4b157a3"
server_key_version: "encryptionkey@0"
I20260923 23:05:34.043629 9181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "2d1c1d324d2249daad78dec390a7fbd6"
format_stamp: "Formatted at 2026-09-23 23:05:34 on dist-test-slave-j4c5"
server_key: "88e655613a841b5f3e986246a85d3573"
server_key_iv: "cf0352c3eb950fac9e30c486b4b157a3"
server_key_version: "encryptionkey@0"
I20260923 23:05:34.045112 9181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:34.045917 9196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:34.046146 9181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:34.046231 9181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "2d1c1d324d2249daad78dec390a7fbd6"
format_stamp: "Formatted at 2026-09-23 23:05:34 on dist-test-slave-j4c5"
server_key: "88e655613a841b5f3e986246a85d3573"
server_key_iv: "cf0352c3eb950fac9e30c486b4b157a3"
server_key_version: "encryptionkey@0"
I20260923 23:05:34.046289 9181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:34.058580 9181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:34.060988 9181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:34.061146 9181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:34.061414 9181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:34.061743 9181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:34.061779 9181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:34.061801 9181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:34.061815 9181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:34.067919 9181 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:42577
I20260923 23:05:34.067987 9309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:42577 every 8 connection(s)
I20260923 23:05:34.068327 9181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[8940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:34.076939 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 9181
I20260923 23:05:34.077080 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:34.077271 5416 external_mini_cluster.cc:1428] Setting key a2cc7f4b10ae317514b2486c82771f59
I20260923 23:05:34.079137 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:34.069177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:60753 (local address 127.5.74.62:40635)
0923 23:05:34.069257 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:34.069259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:34.069459 (+ 200us) server_negotiation.cc:407] Connection header received
0923 23:05:34.070459 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:34.070460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:34.070471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:34.070533 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:34.071164 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.071692 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.072914 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.073124 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.073192 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:34.075511 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:34.075513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:34.075513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:34.075520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:34.077455 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:34.077974 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:34.077976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:34.077976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:34.078043 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:34.078323 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:34.078325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:34.078325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:34.078472 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:34.078550 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:34.078880 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 23:05:34.078991 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 23:05:34.079176 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:34.069152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:40635 (local address 127.5.74.2:60753)
0923 23:05:34.069322 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:34.069366 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:34.070323 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:34.070604 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:34.070607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:34.070875 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:34.071077 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:34.071081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.071782 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.071783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:34.072795 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:34.072798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.073204 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.073205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:34.073302 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:34.073866 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:34.073875 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:34.075361 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:34.077629 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:34.077635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:34.077638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:34.077869 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:34.078119 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:34.078121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:34.078121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:34.078247 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:34.078565 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:34.078571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:34.078728 (+ 157us) client_negotiation.cc:769] Sending connection context
0923 23:05:34.078838 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 23:05:34.078977 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260923 23:05:34.080034 9310 heartbeater.cc:347] Connected to a master server at 127.5.74.62:40635
I20260923 23:05:34.080142 9310 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:34.080377 9310 heartbeater.cc:511] Master 127.5.74.62:40635 requested a full tablet report, sending...
I20260923 23:05:34.080821 8985 ts_manager.cc:194] Registered new tserver with Master: 2d1c1d324d2249daad78dec390a7fbd6 (127.5.74.2:42577)
I20260923 23:05:34.081383 8985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:60753
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:34.109179 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40635
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:34.192035 9315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:34.192312 9315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:34.192340 9315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:34.193470 9315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:34.193506 9315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:34.193533 9315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:34.194989 9315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:34.195194 9315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:34.195390 9315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:34.200834 9321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:34.200850 9323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:34.200918 9320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:34.201243 9315 server_base.cc:1034] running on GCE node
I20260923 23:05:34.201498 9315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:34.201747 9315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:34.202965 9315 hybrid_clock.cc:648] HybridClock initialized: now 1790204734202935 us; error 40 us; skew 500 ppm
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[8940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:34.205922 9315 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:34.206470 9315 webserver.cc:460] Webserver started at http://127.5.74.3:35039/ using document root <none> and password file <none>
I20260923 23:05:34.206717 9315 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:34.206805 9315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:34.206971 9315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:34.208140 9315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "a263ba78f5a54f7bb915c4e4e0d38512"
format_stamp: "Formatted at 2026-09-23 23:05:34 on dist-test-slave-j4c5"
server_key: "325e2cf3a3f16bd5d9ec12590e34056b"
server_key_iv: "576324dcd763b22fd2b8babbc120ef38"
server_key_version: "encryptionkey@0"
I20260923 23:05:34.208629 9315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "a263ba78f5a54f7bb915c4e4e0d38512"
format_stamp: "Formatted at 2026-09-23 23:05:34 on dist-test-slave-j4c5"
server_key: "325e2cf3a3f16bd5d9ec12590e34056b"
server_key_iv: "576324dcd763b22fd2b8babbc120ef38"
server_key_version: "encryptionkey@0"
I20260923 23:05:34.210294 9315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:05:34.211256 9330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:34.211529 9315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:34.211647 9315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "a263ba78f5a54f7bb915c4e4e0d38512"
format_stamp: "Formatted at 2026-09-23 23:05:34 on dist-test-slave-j4c5"
server_key: "325e2cf3a3f16bd5d9ec12590e34056b"
server_key_iv: "576324dcd763b22fd2b8babbc120ef38"
server_key_version: "encryptionkey@0"
I20260923 23:05:34.211733 9315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:34.230870 9315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:34.233306 9315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:34.233505 9315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:34.233736 9315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:34.234066 9315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:34.234120 9315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:34.234162 9315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:34.234190 9315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:34.240814 9315 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:40565
I20260923 23:05:34.240871 9443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:40565 every 8 connection(s)
I20260923 23:05:34.241386 9315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:05:34.244658 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 9315
I20260923 23:05:34.244779 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:34.244983 5416 external_mini_cluster.cc:1428] Setting key 187406d989db41fff3c63873241e2f41
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[8940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204734, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:34.252684 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:34.242260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:50011 (local address 127.5.74.62:40635)
0923 23:05:34.242331 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:34.242333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:34.242594 (+ 261us) server_negotiation.cc:407] Connection header received
0923 23:05:34.243522 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:34.243524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:34.243536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:34.243779 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:34.244539 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.245270 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.246627 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.246832 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.246895 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:34.249389 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:34.249391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:34.249391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:34.249401 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:34.251051 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:34.251522 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:34.251524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:34.251524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:34.251605 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:34.251920 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:34.251922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:34.251922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:34.252059 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:34.252125 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:34.252538 (+ 413us) server_negotiation.cc:299] Negotiation successful
0923 23:05:34.252615 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 23:05:34.252830 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:34.242285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:40635 (local address 127.5.74.3:50011)
0923 23:05:34.242460 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:34.242502 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:34.243413 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:34.243872 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:34.243876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:34.244135 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:34.244429 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:34.244449 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.245490 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.245491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:34.246538 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:34.246540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.246916 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.246917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:34.247024 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:34.247580 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:34.247588 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:34.249243 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:34.251175 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:34.251181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:34.251183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:34.251421 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:34.251703 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:34.251704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:34.251704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:34.251830 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:34.252133 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:34.252138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:34.252383 (+ 245us) client_negotiation.cc:769] Sending connection context
0923 23:05:34.252480 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 23:05:34.252623 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260923 23:05:34.253405 9444 heartbeater.cc:347] Connected to a master server at 127.5.74.62:40635
I20260923 23:05:34.253487 9444 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:34.253705 9444 heartbeater.cc:511] Master 127.5.74.62:40635 requested a full tablet report, sending...
I20260923 23:05:34.254139 8985 ts_manager.cc:194] Registered new tserver with Master: a263ba78f5a54f7bb915c4e4e0d38512 (127.5.74.3:40565)
I20260923 23:05:34.254525 8985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:50011
I20260923 23:05:34.257050 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[8940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[8940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:34.282584 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:34.273462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.5.74.1:36819)
0923 23:05:34.273704 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:34.273709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:34.273722 (+ 13us) server_negotiation.cc:407] Connection header received
0923 23:05:34.273781 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:34.273783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:34.273798 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:34.273879 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:34.274462 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.275281 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.276131 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.276621 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.276691 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:34.279326 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:34.279336 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:34.279336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:34.279344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:34.281369 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:34.281826 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:34.281830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:34.281830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:34.281893 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:34.282075 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:34.282075 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:34.282075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:34.282204 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:34.282305 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:34.282428 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 23:05:34.282500 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":127,"threads_started":1}
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[8940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:34.291817 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:34.283971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.5.74.62:40635)
0923 23:05:34.284022 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:34.284024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:34.284103 (+ 79us) server_negotiation.cc:407] Connection header received
0923 23:05:34.284186 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:34.284187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:34.284197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:34.284262 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:34.284832 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.285356 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.286073 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:34.286267 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:34.286357 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:34.289048 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:34.289050 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:34.289050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:34.289057 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:34.290570 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:34.290972 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:34.290973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:34.290973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:34.291026 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:34.291256 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:34.291258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:34.291258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:34.291433 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:34.291540 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:34.291652 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 23:05:34.291729 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260923 23:05:34.292199 8985 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44746
I20260923 23:05:34.292651 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 9047
I20260923 23:05:34.298422 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 9181
I20260923 23:05:34.302791 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 9315
I20260923 23:05:34.307220 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 8953
2026-09-23T23:05:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1982 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[9456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[9456](info): set up 2 sockets
Sep 23 23:05:34 dist-test-slave-j4c5 krb5kdc[9456](info): commencing operation
krb5kdc: starting...
W20260923 23:05:35.360703 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:35 dist-test-slave-j4c5 krb5kdc[9456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:35.498363 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:43321
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:43321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:35.585129 9469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:35.585346 9469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:35.585371 9469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:35.586583 9469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:35.586628 9469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:35.586642 9469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:35.586655 9469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:35.586668 9469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:35.588241 9469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:43321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:43321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:35.588517 9469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:35.588755 9469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:35.594533 9474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:35.594527 9475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:35.594527 9477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:35.594655 9469 server_base.cc:1034] running on GCE node
I20260923 23:05:35.595373 9469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:35.595783 9469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:35.597088 9469 hybrid_clock.cc:648] HybridClock initialized: now 1790204735597061 us; error 67 us; skew 500 ppm
Sep 23 23:05:35 dist-test-slave-j4c5 krb5kdc[9456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204735, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:35.600077 9469 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:35.600541 9469 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:35.600618 9469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:35.600777 9469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:35.601992 9469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "6a053e7f11ee48099a54f4c9e7be32b7"
format_stamp: "Formatted at 2026-09-23 23:05:35 on dist-test-slave-j4c5"
server_key: "144c022c612e658d4bfb86f193244982"
server_key_iv: "e3e992e968ef53655900fa009dbcb378"
server_key_version: "encryptionkey@0"
I20260923 23:05:35.602376 9469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "6a053e7f11ee48099a54f4c9e7be32b7"
format_stamp: "Formatted at 2026-09-23 23:05:35 on dist-test-slave-j4c5"
server_key: "144c022c612e658d4bfb86f193244982"
server_key_iv: "e3e992e968ef53655900fa009dbcb378"
server_key_version: "encryptionkey@0"
I20260923 23:05:35.603881 9469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:35.604858 9483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:35.605188 9469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:35.605309 9469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "6a053e7f11ee48099a54f4c9e7be32b7"
format_stamp: "Formatted at 2026-09-23 23:05:35 on dist-test-slave-j4c5"
server_key: "144c022c612e658d4bfb86f193244982"
server_key_iv: "e3e992e968ef53655900fa009dbcb378"
server_key_version: "encryptionkey@0"
I20260923 23:05:35.605393 9469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:35.616947 9469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:35.619683 9469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:35.619916 9469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:35.625006 9469 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:43321
I20260923 23:05:35.625025 9535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:43321 every 8 connection(s)
I20260923 23:05:35.625416 9469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:35.626400 9536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:35.628750 9536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7: Bootstrap starting.
I20260923 23:05:35.629318 9536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:35.629556 9536 log.cc:824] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:35.630302 9536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7: No bootstrap required, opened a new log
I20260923 23:05:35.631910 9536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a053e7f11ee48099a54f4c9e7be32b7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43321 } }
I20260923 23:05:35.632056 9536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:35.632079 9536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a053e7f11ee48099a54f4c9e7be32b7, State: Initialized, Role: FOLLOWER
I20260923 23:05:35.632277 9536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a053e7f11ee48099a54f4c9e7be32b7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43321 } }
I20260923 23:05:35.632347 9536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:35.632369 9536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:35.632400 9536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:35.633530 9536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a053e7f11ee48099a54f4c9e7be32b7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43321 } }
I20260923 23:05:35.633642 9536 leader_election.cc:302] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a053e7f11ee48099a54f4c9e7be32b7; no voters:
I20260923 23:05:35.633780 9536 leader_election.cc:288] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:35.633873 9539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:35.634053 9539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [term 1 LEADER]: Becoming Leader. State: Replica: 6a053e7f11ee48099a54f4c9e7be32b7, State: Running, Role: LEADER
I20260923 23:05:35.634068 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 9469
I20260923 23:05:35.634182 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:35.634164 9539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a053e7f11ee48099a54f4c9e7be32b7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43321 } }
I20260923 23:05:35.634192 9536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:35.634370 5416 external_mini_cluster.cc:1428] Setting key 3e6628064b044fa761d1acdbb90e63a8
I20260923 23:05:35.634752 9540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a053e7f11ee48099a54f4c9e7be32b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a053e7f11ee48099a54f4c9e7be32b7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43321 } } }
I20260923 23:05:35.634851 9540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:35.635025 9541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a053e7f11ee48099a54f4c9e7be32b7. Latest consensus state: current_term: 1 leader_uuid: "6a053e7f11ee48099a54f4c9e7be32b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a053e7f11ee48099a54f4c9e7be32b7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43321 } } }
I20260923 23:05:35.635102 9541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:35.635229 9549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:35.635892 9549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:35.637868 9549 catalog_manager.cc:1347] Generated new cluster ID: 675004ba57214e4591de7704976d18b5
I20260923 23:05:35.637939 9549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:05:35 dist-test-slave-j4c5 krb5kdc[9456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:35.648415 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:35.635010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.5.74.62:43321)
0923 23:05:35.635623 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:35.635634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:35.635719 (+ 85us) server_negotiation.cc:407] Connection header received
0923 23:05:35.635881 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:35.635889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:35.635960 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:35.636124 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:35.637021 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:35.638686 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:35.639799 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:35.640428 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:35.640526 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:35.644361 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:35.644368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:35.644368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:35.644385 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:35.646579 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:35.647019 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:35.647022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:35.647023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:35.647148 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:35.647359 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:35.647360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:35.647360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:35.647703 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:35.647878 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:35.647949 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 23:05:35.648096 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":76,"threads_started":1}
I20260923 23:05:35.655305 9549 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:35.655969 9549 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:35.665292 9549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7: Generated new TSK 0
I20260923 23:05:35.665529 9549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:35.708928 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43321
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:35.819233 9562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:35.819461 9562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:35.819487 9562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:35.820775 9562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:35.820827 9562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:35.820856 9562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:35.822422 9562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:35.822685 9562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:35.822950 9562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:35.828059 9568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:35.828082 9567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:35.828061 9570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:35.828364 9562 server_base.cc:1034] running on GCE node
I20260923 23:05:35.828627 9562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:35.829064 9562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:35.830288 9562 hybrid_clock.cc:648] HybridClock initialized: now 1790204735830244 us; error 87 us; skew 500 ppm
Sep 23 23:05:35 dist-test-slave-j4c5 krb5kdc[9456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204735, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:35.833091 9562 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:35.833478 9562 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:35.833526 9562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:35.833657 9562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:35.834753 9562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "7b3745ffbb684b8d969a13587cdeaf96"
format_stamp: "Formatted at 2026-09-23 23:05:35 on dist-test-slave-j4c5"
server_key: "31c3aa6a299e0cc0cbd82b73a7cacd9a"
server_key_iv: "acbec044130271139cbca42372b9b3c0"
server_key_version: "encryptionkey@0"
I20260923 23:05:35.835108 9562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "7b3745ffbb684b8d969a13587cdeaf96"
format_stamp: "Formatted at 2026-09-23 23:05:35 on dist-test-slave-j4c5"
server_key: "31c3aa6a299e0cc0cbd82b73a7cacd9a"
server_key_iv: "acbec044130271139cbca42372b9b3c0"
server_key_version: "encryptionkey@0"
I20260923 23:05:35.836510 9562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:05:35.837450 9576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:35.837699 9562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:35.837824 9562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "7b3745ffbb684b8d969a13587cdeaf96"
format_stamp: "Formatted at 2026-09-23 23:05:35 on dist-test-slave-j4c5"
server_key: "31c3aa6a299e0cc0cbd82b73a7cacd9a"
server_key_iv: "acbec044130271139cbca42372b9b3c0"
server_key_version: "encryptionkey@0"
I20260923 23:05:35.837904 9562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:35.854743 9562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:35.857297 9562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:35.857482 9562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:35.857791 9562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:35.858143 9562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:35.858207 9562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:35.858246 9562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:35.858281 9562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:35.865949 9562 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:36749
I20260923 23:05:35.866037 9689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:36749 every 8 connection(s)
I20260923 23:05:35.866290 9562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:05:35 dist-test-slave-j4c5 krb5kdc[9456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204735, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:35.875792 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 9562
I20260923 23:05:35.875939 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:35.876210 5416 external_mini_cluster.cc:1428] Setting key 1be9804003b426eae1f201598de0e7b0
I20260923 23:05:35.876924 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:35.867161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:60167 (local address 127.5.74.62:43321)
0923 23:05:35.867210 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:35.867212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:35.867435 (+ 223us) server_negotiation.cc:407] Connection header received
0923 23:05:35.867985 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:35.867986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:35.868003 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:35.868057 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:35.869056 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:35.869588 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:35.870601 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:35.870823 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:35.870911 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:35.873290 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:35.873292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:35.873293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:35.873305 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:35.874975 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:35.875522 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:35.875524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:35.875524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:35.875615 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:35.875943 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:35.875944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:35.875944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:35.876122 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:35.876247 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:35.876707 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 23:05:35.876798 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:35.877103 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:35.867133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:43321 (local address 127.5.74.1:60167)
0923 23:05:35.867301 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:35.867340 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:35.867872 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:35.868124 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:35.868128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:35.868445 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:35.868947 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:35.868952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:35.869685 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:35.869686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:35.870504 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:35.870506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:35.870923 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:35.870924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:35.871058 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:35.871614 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:35.871623 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:35.873184 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:35.875118 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:35.875127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:35.875130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:35.875413 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:35.875708 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:35.875709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:35.875709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:35.875836 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:35.876240 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:35.876247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:35.876514 (+ 267us) client_negotiation.cc:769] Sending connection context
0923 23:05:35.876661 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 23:05:35.876846 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260923 23:05:35.877698 9690 heartbeater.cc:347] Connected to a master server at 127.5.74.62:43321
I20260923 23:05:35.877856 9690 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:35.878037 9690 heartbeater.cc:511] Master 127.5.74.62:43321 requested a full tablet report, sending...
I20260923 23:05:35.878672 9500 ts_manager.cc:194] Registered new tserver with Master: 7b3745ffbb684b8d969a13587cdeaf96 (127.5.74.1:36749)
I20260923 23:05:35.879292 9500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:60167
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:35.904330 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43321
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:35.991267 9695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:35.991489 9695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:35.991518 9695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:35.992868 9695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:35.992911 9695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:35.992939 9695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:35.994565 9695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:35.994823 9695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:35.995055 9695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:36.000257 9700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.000311 9703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.000308 9701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:36.000536 9695 server_base.cc:1034] running on GCE node
I20260923 23:05:36.000751 9695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:36.000959 9695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:36.002136 9695 hybrid_clock.cc:648] HybridClock initialized: now 1790204736002123 us; error 31 us; skew 500 ppm
Sep 23 23:05:36 dist-test-slave-j4c5 krb5kdc[9456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204736, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:36.004961 9695 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:36.005362 9695 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:36.005414 9695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:36.005507 9695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:36.006546 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "ccfda37b6bf94b21833b19f9bc2be37d"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "cd05eace0d35ee2ee334f7f3b2dccbe6"
server_key_iv: "1f5ff67643765f80ddd0e123ccf2022d"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.006856 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "ccfda37b6bf94b21833b19f9bc2be37d"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "cd05eace0d35ee2ee334f7f3b2dccbe6"
server_key_iv: "1f5ff67643765f80ddd0e123ccf2022d"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.008353 9695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:05:36.009369 9709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.009677 9695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:36.009786 9695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "ccfda37b6bf94b21833b19f9bc2be37d"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "cd05eace0d35ee2ee334f7f3b2dccbe6"
server_key_iv: "1f5ff67643765f80ddd0e123ccf2022d"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.009874 9695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:36.025728 9695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:36.028105 9695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:36.028291 9695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:36.028532 9695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:36.028844 9695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:36.028904 9695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.028939 9695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:36.028968 9695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.035032 9695 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:36055
I20260923 23:05:36.035094 9822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:36055 every 8 connection(s)
I20260923 23:05:36.035380 9695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:05:36.039925 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 9695
I20260923 23:05:36.040048 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:36.040310 5416 external_mini_cluster.cc:1428] Setting key e72fc0e4271fc404c91eddd998f6e1cc
Sep 23 23:05:36 dist-test-slave-j4c5 krb5kdc[9456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204736, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:36.045464 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.036205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:58089 (local address 127.5.74.62:43321)
0923 23:05:36.036253 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.036256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.036485 (+ 229us) server_negotiation.cc:407] Connection header received
0923 23:05:36.036957 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.036958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.036968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.037021 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.037911 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.038436 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.039434 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.039637 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.039695 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.042142 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.042144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.042145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:36.042153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.043715 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:36.044247 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:36.044248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:36.044249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:36.044317 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:36.044701 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:36.044702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:36.044702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:36.044811 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.044865 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.045267 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.045361 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:05:36.045697 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.036159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:43321 (local address 127.5.74.2:58089)
0923 23:05:36.036340 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.036375 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.036866 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.037070 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.037073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.037325 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:36.037827 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.037832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.038519 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.038519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.039307 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.039309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.039712 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.039713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.039808 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.040441 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:36.040451 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:36.042028 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:36.043802 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:36.043809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:36.043811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:36.044130 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:36.044392 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:36.044393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:36.044394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:36.044551 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:36.044880 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:36.044887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:36.045063 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.045186 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.045395 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260923 23:05:36.046237 9823 heartbeater.cc:347] Connected to a master server at 127.5.74.62:43321
I20260923 23:05:36.046326 9823 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:36.046687 9823 heartbeater.cc:511] Master 127.5.74.62:43321 requested a full tablet report, sending...
I20260923 23:05:36.047207 9500 ts_manager.cc:194] Registered new tserver with Master: ccfda37b6bf94b21833b19f9bc2be37d (127.5.74.2:36055)
I20260923 23:05:36.047736 9500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:58089
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:36.068295 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43321
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:36.154860 9828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:36.155102 9828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:36.155128 9828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:36.156519 9828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:36.156574 9828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:36.156605 9828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:36.158264 9828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:36.158528 9828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:36.158790 9828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:36.164041 9834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.164045 9833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.164080 9836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:36.164499 9828 server_base.cc:1034] running on GCE node
I20260923 23:05:36.164794 9828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:36.165093 9828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:36.166289 9828 hybrid_clock.cc:648] HybridClock initialized: now 1790204736166273 us; error 32 us; skew 500 ppm
Sep 23 23:05:36 dist-test-slave-j4c5 krb5kdc[9456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204736, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:36.169350 9828 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:36.169754 9828 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:36.169896 9828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:36.170068 9828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:36.171240 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "b65d862a9d904370873dab9e8a74507b"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "0371f97c1f5bf5bbe2aad1d50c96f952"
server_key_iv: "a196c85c344deaeb5b6fc4c726e1a8da"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.171605 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "b65d862a9d904370873dab9e8a74507b"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "0371f97c1f5bf5bbe2aad1d50c96f952"
server_key_iv: "a196c85c344deaeb5b6fc4c726e1a8da"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.173105 9828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:05:36.174054 9842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.174329 9828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:36.174603 9828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "b65d862a9d904370873dab9e8a74507b"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "0371f97c1f5bf5bbe2aad1d50c96f952"
server_key_iv: "a196c85c344deaeb5b6fc4c726e1a8da"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.174710 9828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:36.185913 9828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:36.188347 9828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:36.188522 9828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:36.188782 9828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:36.189107 9828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:36.189162 9828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.189203 9828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:36.189253 9828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.195408 9828 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:44171
I20260923 23:05:36.195698 9955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:44171 every 8 connection(s)
I20260923 23:05:36.195749 9828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:05:36 dist-test-slave-j4c5 krb5kdc[9456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204736, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:36.204028 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 9828
I20260923 23:05:36.204182 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:36.204502 5416 external_mini_cluster.cc:1428] Setting key 295bd3563571df91c880fbff26bcd378
I20260923 23:05:36.207187 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.196557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:35335 (local address 127.5.74.62:43321)
0923 23:05:36.196622 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.196624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.196757 (+ 133us) server_negotiation.cc:407] Connection header received
0923 23:05:36.197301 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.197303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.197316 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.197391 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.198486 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.199156 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.200339 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.200557 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.200617 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.203465 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.203466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.203467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:36.203476 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.205633 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:36.206137 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:36.206139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:36.206139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:36.206272 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:36.206524 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:36.206525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:36.206525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:36.206662 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.206733 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.207043 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.207119 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:05:36.207549 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.196458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:43321 (local address 127.5.74.3:35335)
0923 23:05:36.196617 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.196660 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.197149 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.197507 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.197511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.197779 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:36.198286 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.198292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.199259 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.199260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.200115 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.200117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.200668 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.200669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.200794 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.201532 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:36.201542 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:36.203332 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:36.205755 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:36.205771 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:36.205773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:36.206057 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:36.206347 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:36.206347 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:36.206348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:36.206458 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:36.206756 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:36.206764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:36.206915 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.207010 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.207157 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260923 23:05:36.208011 9956 heartbeater.cc:347] Connected to a master server at 127.5.74.62:43321
I20260923 23:05:36.208108 9956 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:36.208304 9956 heartbeater.cc:511] Master 127.5.74.62:43321 requested a full tablet report, sending...
I20260923 23:05:36.208717 9500 ts_manager.cc:194] Registered new tserver with Master: b65d862a9d904370873dab9e8a74507b (127.5.74.3:44171)
I20260923 23:05:36.209065 9500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:35335
I20260923 23:05:36.216821 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:05:36.225612 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.218518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.5.74.62:43321)
0923 23:05:36.218593 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.218595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.218606 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:36.218633 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.218634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.218643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.218707 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.219371 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.219873 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.220709 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.220952 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.221018 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.222345 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.222346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.222346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:36.222354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.224033 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:36.224663 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:36.224665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:36.224665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:36.224730 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:36.225029 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:36.225030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:36.225030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:36.225219 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.225364 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.225470 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.225542 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:05:36.227604 9500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:36.229408 9500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:36.238225 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.232557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.5.74.3:44171)
0923 23:05:36.232749 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.232754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.232797 (+ 43us) server_negotiation.cc:407] Connection header received
0923 23:05:36.232912 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.233000 (+ 88us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.233020 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.233080 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:36.233322 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.234548 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.236587 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.237345 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.237421 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.237486 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.238079 (+ 593us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.238128 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260923 23:05:36.238233 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.232295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:44171 (local address 127.0.0.1:38606)
0923 23:05:36.232739 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.232749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.232824 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.233205 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.233205 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.233207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:36.233244 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.233244 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.234671 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.234672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.236489 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.236492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.237742 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.237743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.237830 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.237847 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.238042 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.238117 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":23,"thread_start_us":55,"threads_started":1}
I20260923 23:05:36.238520 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.232670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.5.74.1:36749)
0923 23:05:36.232956 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.232960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.232969 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:36.233009 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.233012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.233025 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.233099 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:36.234724 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.235694 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.236951 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.237793 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.237893 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.238032 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.238275 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.238323 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260923 23:05:36.238596 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.232293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:36749 (local address 127.0.0.1:55476)
0923 23:05:36.232815 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.232826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.232873 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.233185 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.233186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.233188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:36.233242 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.233244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.235837 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.235839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.236817 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.236820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.238078 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.238079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.238136 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.238139 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.238193 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.238223 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":22,"thread_start_us":100,"threads_started":1}
I20260923 23:05:36.238884 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.232293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:36055 (local address 127.0.0.1:57468)
0923 23:05:36.232492 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.232537 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.232611 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.232924 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.232927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.232930 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:36.233000 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.233002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.234566 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.234568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.236489 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.236492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.238539 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.238540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.238654 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.238657 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.238705 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.238758 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":254,"thread_start_us":113,"threads_started":1}
I20260923 23:05:36.239125 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.232481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.5.74.2:36055)
0923 23:05:36.232647 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.232653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.232670 (+ 17us) server_negotiation.cc:407] Connection header received
0923 23:05:36.232721 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.232723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.232740 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.232829 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:36.233085 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.234448 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.237585 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.238359 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.238440 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.238505 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.238997 (+ 492us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.239044 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260923 23:05:36.239475 9624 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3b13f1968644068fd4ef974f0e7aa6 (DEFAULT_TABLE table=test-table [id=89c414de65ce437bb5e8d9309e42fe1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:36.239629 9890 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3b13f1968644068fd4ef974f0e7aa6 (DEFAULT_TABLE table=test-table [id=89c414de65ce437bb5e8d9309e42fe1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:36.239773 9624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3b13f1968644068fd4ef974f0e7aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:36.239759 9757 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3b13f1968644068fd4ef974f0e7aa6 (DEFAULT_TABLE table=test-table [id=89c414de65ce437bb5e8d9309e42fe1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:36.239948 9890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3b13f1968644068fd4ef974f0e7aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:36.240029 9757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3b13f1968644068fd4ef974f0e7aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:36.242624 9973 tablet_bootstrap.cc:492] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96: Bootstrap starting.
I20260923 23:05:36.243170 9973 tablet_bootstrap.cc:654] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:36.243813 9973 log.cc:824] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:36.244140 9975 tablet_bootstrap.cc:492] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: Bootstrap starting.
I20260923 23:05:36.244606 9973 tablet_bootstrap.cc:492] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96: No bootstrap required, opened a new log
I20260923 23:05:36.244664 9973 ts_tablet_manager.cc:1397] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:36.244807 9975 tablet_bootstrap.cc:654] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:36.245309 9975 log.cc:824] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:36.245677 9974 tablet_bootstrap.cc:492] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d: Bootstrap starting.
I20260923 23:05:36.246143 9975 tablet_bootstrap.cc:492] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: No bootstrap required, opened a new log
I20260923 23:05:36.246241 9974 tablet_bootstrap.cc:654] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:36.246243 9975 ts_tablet_manager.cc:1397] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:05:36.246543 9973 raft_consensus.cc:348] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } }
I20260923 23:05:36.246742 9973 raft_consensus.cc:374] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:36.246728 9974 log.cc:824] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:36.246779 9973 raft_consensus.cc:729] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3745ffbb684b8d969a13587cdeaf96, State: Initialized, Role: FOLLOWER
I20260923 23:05:36.246991 9973 consensus_queue.cc:260] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } }
I20260923 23:05:36.247331 9973 ts_tablet_manager.cc:1428] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:05:36.247464 9690 heartbeater.cc:503] Master 127.5.74.62:43321 was elected leader, sending a full tablet report...
I20260923 23:05:36.247709 9974 tablet_bootstrap.cc:492] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d: No bootstrap required, opened a new log
I20260923 23:05:36.247816 9974 ts_tablet_manager.cc:1397] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:05:36.248495 9975 raft_consensus.cc:348] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } }
I20260923 23:05:36.248793 9975 raft_consensus.cc:374] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:36.248854 9975 raft_consensus.cc:729] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65d862a9d904370873dab9e8a74507b, State: Initialized, Role: FOLLOWER
I20260923 23:05:36.248991 9975 consensus_queue.cc:260] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } }
I20260923 23:05:36.249312 9975 ts_tablet_manager.cc:1428] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260923 23:05:36.249334 9956 heartbeater.cc:503] Master 127.5.74.62:43321 was elected leader, sending a full tablet report...
I20260923 23:05:36.250240 9974 raft_consensus.cc:348] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } }
I20260923 23:05:36.250427 9974 raft_consensus.cc:374] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:36.250478 9974 raft_consensus.cc:729] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccfda37b6bf94b21833b19f9bc2be37d, State: Initialized, Role: FOLLOWER
I20260923 23:05:36.250602 9974 consensus_queue.cc:260] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } }
I20260923 23:05:36.250901 9974 ts_tablet_manager.cc:1428] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:05:36.250943 9823 heartbeater.cc:503] Master 127.5.74.62:43321 was elected leader, sending a full tablet report...
I20260923 23:05:36.277591 9980 raft_consensus.cc:482] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:36.277732 9980 raft_consensus.cc:504] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } }
I20260923 23:05:36.278223 9980 leader_election.cc:288] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccfda37b6bf94b21833b19f9bc2be37d (127.5.74.2:36055), 7b3745ffbb684b8d969a13587cdeaf96 (127.5.74.1:36749)
I20260923 23:05:36.281847 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.278431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:36749 (local address 127.5.74.3:39895)
0923 23:05:36.278499 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.278512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.278559 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.278872 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.278874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.278876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:36.278953 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.278955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.279773 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.279774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.280918 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.280921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.281528 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.281529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.281617 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.281621 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.281687 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.281731 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":199}
I20260923 23:05:36.281841 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.278626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:39895 (local address 127.5.74.1:36749)
0923 23:05:36.278710 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.278712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.278722 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:36.278754 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.278754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.278780 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.278841 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:36.279063 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.279655 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.281022 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.281429 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.281494 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.281505 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.281710 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.281743 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 23:05:36.282241 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.278535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:57667 (local address 127.5.74.2:36055)
0923 23:05:36.278603 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.278605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.278869 (+ 264us) server_negotiation.cc:407] Connection header received
0923 23:05:36.278898 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.278898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.278911 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.278980 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:36.279246 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.279897 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.281184 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.281733 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.281801 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.281811 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.282107 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.282147 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 23:05:36.282279 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.278439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:36055 (local address 127.5.74.3:57667)
0923 23:05:36.278729 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.278739 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.278795 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.279063 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.279064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.279066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:36.279130 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.279132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.280019 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.280019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.280994 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:36.280996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.281820 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.281821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:36.281902 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.281906 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.281970 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.282016 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":105,"thread_start_us":131,"threads_started":1}
I20260923 23:05:36.282294 9644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3b13f1968644068fd4ef974f0e7aa6" candidate_uuid: "b65d862a9d904370873dab9e8a74507b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3745ffbb684b8d969a13587cdeaf96" is_pre_election: true
I20260923 23:05:36.282500 9644 raft_consensus.cc:2454] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65d862a9d904370873dab9e8a74507b in term 0.
I20260923 23:05:36.282675 9777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3b13f1968644068fd4ef974f0e7aa6" candidate_uuid: "b65d862a9d904370873dab9e8a74507b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" is_pre_election: true
I20260923 23:05:36.282757 9845 leader_election.cc:302] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3745ffbb684b8d969a13587cdeaf96, b65d862a9d904370873dab9e8a74507b; no voters:
I20260923 23:05:36.282891 9777 raft_consensus.cc:2454] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65d862a9d904370873dab9e8a74507b in term 0.
I20260923 23:05:36.282888 9980 raft_consensus.cc:2781] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:36.282969 9980 raft_consensus.cc:482] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:36.283005 9980 raft_consensus.cc:3037] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:36.283779 9980 raft_consensus.cc:504] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } }
I20260923 23:05:36.283970 9980 leader_election.cc:288] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [CANDIDATE]: Term 1 election: Requested vote from peers ccfda37b6bf94b21833b19f9bc2be37d (127.5.74.2:36055), 7b3745ffbb684b8d969a13587cdeaf96 (127.5.74.1:36749)
I20260923 23:05:36.284097 9777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3b13f1968644068fd4ef974f0e7aa6" candidate_uuid: "b65d862a9d904370873dab9e8a74507b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccfda37b6bf94b21833b19f9bc2be37d"
I20260923 23:05:36.284229 9777 raft_consensus.cc:3037] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:36.284427 9644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3b13f1968644068fd4ef974f0e7aa6" candidate_uuid: "b65d862a9d904370873dab9e8a74507b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3745ffbb684b8d969a13587cdeaf96"
I20260923 23:05:36.284528 9644 raft_consensus.cc:3037] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:36.285082 9777 raft_consensus.cc:2454] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65d862a9d904370873dab9e8a74507b in term 1.
I20260923 23:05:36.285269 9843 leader_election.cc:302] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65d862a9d904370873dab9e8a74507b, ccfda37b6bf94b21833b19f9bc2be37d; no voters:
I20260923 23:05:36.285391 9644 raft_consensus.cc:2454] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65d862a9d904370873dab9e8a74507b in term 1.
I20260923 23:05:36.285411 9980 raft_consensus.cc:2781] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:36.285619 9980 raft_consensus.cc:686] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 1 LEADER]: Becoming Leader. State: Replica: b65d862a9d904370873dab9e8a74507b, State: Running, Role: LEADER
I20260923 23:05:36.285822 9980 consensus_queue.cc:237] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } }
W20260923 23:05:36.286033 9824 tablet.cc:2369] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:36.286962 9498 catalog_manager.cc:5495] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b reported cstate change: term changed from 0 to 1, leader changed from <none> to b65d862a9d904370873dab9e8a74507b (127.5.74.3). New cstate: current_term: 1 leader_uuid: "b65d862a9d904370873dab9e8a74507b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65d862a9d904370873dab9e8a74507b" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44171 } health_report { overall_health: HEALTHY } } }
I20260923 23:05:36.300165 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.297456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.5.74.3:44171)
0923 23:05:36.297573 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.297574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.297588 (+ 14us) server_negotiation.cc:407] Connection header received
0923 23:05:36.297620 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.297621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.297640 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.297709 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:36.297882 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.298458 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.299268 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:36.299486 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:36.299551 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:36.299723 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:36.299832 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:36.299899 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.299989 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.300008 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 23:05:36.302317 9777 raft_consensus.cc:1261] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [term 1 FOLLOWER]: Refusing update from remote peer b65d862a9d904370873dab9e8a74507b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:36.302320 9644 raft_consensus.cc:1261] T 1d3b13f1968644068fd4ef974f0e7aa6 P 7b3745ffbb684b8d969a13587cdeaf96 [term 1 FOLLOWER]: Refusing update from remote peer b65d862a9d904370873dab9e8a74507b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:36.302734 9983 consensus_queue.cc:1035] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccfda37b6bf94b21833b19f9bc2be37d" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 36055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:36.302935 9980 consensus_queue.cc:1035] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3745ffbb684b8d969a13587cdeaf96" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:36.305088 9986 mvcc.cc:205] Tried to move back new op lower bound from 7332678599891116032 to 7332678599827767296. Current Snapshot: MvccSnapshot[applied={T|T < 7332678599891116032}]
I20260923 23:05:36.305657 9987 mvcc.cc:205] Tried to move back new op lower bound from 7332678599891116032 to 7332678599827767296. Current Snapshot: MvccSnapshot[applied={T|T < 7332678599891116032}]
I20260923 23:05:36.307313 9988 mvcc.cc:205] Tried to move back new op lower bound from 7332678599891116032 to 7332678599827767296. Current Snapshot: MvccSnapshot[applied={T|T < 7332678599891116032 or (T in {7332678599891116032})}]
I20260923 23:05:36.307623 9498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:36.307724 9498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:36.308648 9498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a053e7f11ee48099a54f4c9e7be32b7: Sending DeleteTablet for 3 replicas of tablet 1d3b13f1968644068fd4ef974f0e7aa6
I20260923 23:05:36.309038 9757 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3b13f1968644068fd4ef974f0e7aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:36 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:57468
I20260923 23:05:36.309101 9624 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3b13f1968644068fd4ef974f0e7aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:36 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:55476
I20260923 23:05:36.309247 9890 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3b13f1968644068fd4ef974f0e7aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:36 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:38606
I20260923 23:05:36.309295 9996 tablet_replica.cc:330] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d: stopping tablet replica
I20260923 23:05:36.309396 9996 raft_consensus.cc:2229] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:36.309504 9996 raft_consensus.cc:2258] T 1d3b13f1968644068fd4ef974f0e7aa6 P ccfda37b6bf94b21833b19f9bc2be37d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:36.309616 9998 tablet_replica.cc:330] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: stopping tablet replica
I20260923 23:05:36.309746 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 9562
I20260923 23:05:36.309798 9998 raft_consensus.cc:2229] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:36.310583 9998 raft_consensus.cc:2258] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:36.311661 9998 ts_tablet_manager.cc:1905] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:36.313715 9998 ts_tablet_manager.cc:1918] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:36.313799 9998 log.cc:1196] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/1d3b13f1968644068fd4ef974f0e7aa6
I20260923 23:05:36.314109 9998 ts_tablet_manager.cc:1939] T 1d3b13f1968644068fd4ef974f0e7aa6 P b65d862a9d904370873dab9e8a74507b: Deleting consensus metadata
I20260923 23:05:36.314929 9485 catalog_manager.cc:4841] TS b65d862a9d904370873dab9e8a74507b (127.5.74.3:44171): tablet 1d3b13f1968644068fd4ef974f0e7aa6 (table test-table [id=89c414de65ce437bb5e8d9309e42fe1e]) successfully deleted
W20260923 23:05:36.316248 9486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:36749 (error 108)
I20260923 23:05:36.316376 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 9695
I20260923 23:05:36.320730 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.320524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:36749 (local address 127.0.0.1:55486)
0923 23:05:36.320619 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.320678 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:36749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 23:05:36.320906 9486 catalog_manager.cc:4584] TS 7b3745ffbb684b8d969a13587cdeaf96 (127.5.74.1:36749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d3b13f1968644068fd4ef974f0e7aa6: Network error: Client connection negotiation failed: client connection to 127.5.74.1:36749: connect: Connection refused (error 111)
I20260923 23:05:36.322815 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.322683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:36055 (local address 127.0.0.1:57476)
0923 23:05:36.322746 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.322778 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.2:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 23:05:36.322844 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 9828
I20260923 23:05:36.327921 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 9469
2026-09-23T23:05:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2020 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T23:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:36Z Disabled control of system clock
I20260923 23:05:36.357084 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34041
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:34041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 23:05:36.441581 10003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:36.441819 10003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:36.441843 10003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:36.443019 10003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:36.443066 10003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:36.443081 10003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:36.443096 10003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:36.443152 10003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:36.444535 10003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:34041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:36.444823 10003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:36.445142 10003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:36.450114 10009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.450107 10011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.450103 10008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:36.450541 10003 server_base.cc:1034] running on GCE node
I20260923 23:05:36.450800 10003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:36.451063 10003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:36.452217 10003 hybrid_clock.cc:648] HybridClock initialized: now 1790204736452197 us; error 40 us; skew 500 ppm
I20260923 23:05:36.453513 10003 webserver.cc:460] Webserver started at http://127.5.74.62:42841/ using document root <none> and password file <none>
I20260923 23:05:36.453763 10003 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:36.453831 10003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:36.453982 10003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:36.455076 10003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "ec624570228745a78b8e276d75c351bc"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "4dd44ce8d6f54ea5fcbbfa9ab34889c3"
server_key_iv: "5716599a11a6c7fa96aba109934993d1"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.455459 10003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "ec624570228745a78b8e276d75c351bc"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "4dd44ce8d6f54ea5fcbbfa9ab34889c3"
server_key_iv: "5716599a11a6c7fa96aba109934993d1"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.456885 10003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:05:36.457666 10017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.457883 10003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:36.457980 10003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "ec624570228745a78b8e276d75c351bc"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "4dd44ce8d6f54ea5fcbbfa9ab34889c3"
server_key_iv: "5716599a11a6c7fa96aba109934993d1"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.458060 10003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:36.461479 10003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:36.461747 10003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:36.461884 10003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:36.465713 10003 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:34041
I20260923 23:05:36.465746 10069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:34041 every 8 connection(s)
I20260923 23:05:36.466094 10003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:36.466904 10070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:36.470333 10070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc: Bootstrap starting.
I20260923 23:05:36.470970 10070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:36.471263 10070 log.cc:824] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:36.471597 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 10003
I20260923 23:05:36.471731 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:36.471922 5416 external_mini_cluster.cc:1428] Setting key 67fe66c2fcdf648fd691d0b09962a3e9
I20260923 23:05:36.472102 10070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc: No bootstrap required, opened a new log
I20260923 23:05:36.474044 10070 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec624570228745a78b8e276d75c351bc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34041 } }
I20260923 23:05:36.474220 10070 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:36.474267 10070 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec624570228745a78b8e276d75c351bc, State: Initialized, Role: FOLLOWER
I20260923 23:05:36.474428 10070 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec624570228745a78b8e276d75c351bc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34041 } }
I20260923 23:05:36.474802 10070 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:36.474877 10070 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:36.474937 10070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:36.475822 10070 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec624570228745a78b8e276d75c351bc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34041 } }
I20260923 23:05:36.475952 10070 leader_election.cc:302] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec624570228745a78b8e276d75c351bc; no voters:
I20260923 23:05:36.475999 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.472605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.5.74.62:34041)
0923 23:05:36.472794 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.472802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.472828 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:05:36.474026 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.474035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.474080 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.474337 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.475037 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.475042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.475042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:36.475047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.475092 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:36.475109 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.475198 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.475343 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.475405 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260923 23:05:36.476121 10070 leader_election.cc:288] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:36.476337 10076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:36.476428 10070 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:36.476493 10076 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [term 1 LEADER]: Becoming Leader. State: Replica: ec624570228745a78b8e276d75c351bc, State: Running, Role: LEADER
I20260923 23:05:36.476583 10076 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec624570228745a78b8e276d75c351bc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34041 } }
I20260923 23:05:36.476928 10076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec624570228745a78b8e276d75c351bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec624570228745a78b8e276d75c351bc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34041 } } }
I20260923 23:05:36.477012 10076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:36.477152 10076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec624570228745a78b8e276d75c351bc. Latest consensus state: current_term: 1 leader_uuid: "ec624570228745a78b8e276d75c351bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec624570228745a78b8e276d75c351bc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34041 } } }
I20260923 23:05:36.477234 10076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:36.477311 10083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:36.477882 10083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:36.479492 10083 catalog_manager.cc:1347] Generated new cluster ID: 1c1ecb8d3fcc4e7daed98678ad1bf7b9
I20260923 23:05:36.479549 10083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:36.492107 10083 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:36.492324 10083 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:36.498257 10083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc: Generated new TSK 0
I20260923 23:05:36.498555 10083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:05:36.505201 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34041
--builtin_ntp_servers=127.5.74.20:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 23:05:36.610287 10094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:36.610608 10094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:36.610641 10094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:36.612005 10094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:36.612054 10094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:36.612084 10094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:36.613595 10094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:36.613876 10094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:36.614199 10094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:36.620702 10100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.620868 10102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:36.620908 10094 server_base.cc:1034] running on GCE node
W20260923 23:05:36.620579 10099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:36.621182 10094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:36.621536 10094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:36.622785 10094 hybrid_clock.cc:648] HybridClock initialized: now 1790204736622757 us; error 41 us; skew 500 ppm
I20260923 23:05:36.624537 10094 webserver.cc:460] Webserver started at http://127.5.74.1:44435/ using document root <none> and password file <none>
I20260923 23:05:36.624832 10094 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:36.624884 10094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:36.625028 10094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:36.626281 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "43a565ccc8ba4edc9778785d7c484dd1"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "b6038e31fb7de2c3912681e334e4cf22"
server_key_iv: "f1568ff5886f82b4e28ab704abe4fc74"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.626701 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "43a565ccc8ba4edc9778785d7c484dd1"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "b6038e31fb7de2c3912681e334e4cf22"
server_key_iv: "f1568ff5886f82b4e28ab704abe4fc74"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.628206 10094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260923 23:05:36.629251 10108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.629442 10094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:36.629540 10094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "43a565ccc8ba4edc9778785d7c484dd1"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "b6038e31fb7de2c3912681e334e4cf22"
server_key_iv: "f1568ff5886f82b4e28ab704abe4fc74"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.629622 10094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:36.633409 10094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:36.633708 10094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:36.633867 10094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:36.634116 10094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:36.634512 10094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:36.634550 10094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.634572 10094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:36.634616 10094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.642169 10094 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:45899
I20260923 23:05:36.642198 10221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:45899 every 8 connection(s)
I20260923 23:05:36.642644 10094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:36.646536 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.643713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:33687 (local address 127.5.74.62:34041)
0923 23:05:36.643811 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.643814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.644039 (+ 225us) server_negotiation.cc:407] Connection header received
0923 23:05:36.644718 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.644720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.644727 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.644806 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.646201 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.646204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.646204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:36.646210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.646229 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:36.646234 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.646328 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.646440 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.646458 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 23:05:36.646770 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.643688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34041 (local address 127.5.74.1:33687)
0923 23:05:36.643905 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.643943 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.644576 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.644887 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.644891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.644902 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:36.646011 (+ 1109us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:05:36.646012 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:36.646046 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:05:36.646082 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:36.646323 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:36.646329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:36.646333 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.646437 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.646507 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260923 23:05:36.647296 10222 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34041
I20260923 23:05:36.647415 10222 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:36.647583 10222 heartbeater.cc:511] Master 127.5.74.62:34041 requested a full tablet report, sending...
I20260923 23:05:36.648180 10034 ts_manager.cc:194] Registered new tserver with Master: 43a565ccc8ba4edc9778785d7c484dd1 (127.5.74.1:45899)
I20260923 23:05:36.651254 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 10094
I20260923 23:05:36.651391 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:36.651603 5416 external_mini_cluster.cc:1428] Setting key 9c29a41bd157c8e9bb0cabc91ecee508
I20260923 23:05:36.652537 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34041
--builtin_ntp_servers=127.5.74.20:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 23:05:36.739277 10225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:36.739509 10225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:36.739535 10225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:36.740878 10225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:36.740928 10225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:36.740957 10225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:36.742399 10225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:36.742688 10225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:36.743029 10225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:36.748368 10233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.748428 10230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.748440 10231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:36.748576 10225 server_base.cc:1034] running on GCE node
I20260923 23:05:36.748916 10225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:36.749187 10225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:36.750367 10225 hybrid_clock.cc:648] HybridClock initialized: now 1790204736750329 us; error 47 us; skew 500 ppm
I20260923 23:05:36.751717 10225 webserver.cc:460] Webserver started at http://127.5.74.2:41499/ using document root <none> and password file <none>
I20260923 23:05:36.751992 10225 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:36.752043 10225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:36.752240 10225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:36.753316 10225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "1045643f73404477bc4f069d7abed9a0"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "f3f5f2b8501a528a2595364a4ad59bba"
server_key_iv: "3421ddccd6c84fc4faf204d719b907ef"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.753675 10225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "1045643f73404477bc4f069d7abed9a0"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "f3f5f2b8501a528a2595364a4ad59bba"
server_key_iv: "3421ddccd6c84fc4faf204d719b907ef"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.755124 10225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:36.756038 10239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.756269 10225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:36.756397 10225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "1045643f73404477bc4f069d7abed9a0"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "f3f5f2b8501a528a2595364a4ad59bba"
server_key_iv: "3421ddccd6c84fc4faf204d719b907ef"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.756587 10225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:36.760318 10225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:36.760653 10225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:36.760813 10225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:36.761024 10225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:36.761380 10225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:36.761442 10225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.761484 10225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:36.761512 10225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.768018 10225 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:32933
I20260923 23:05:36.768064 10352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:32933 every 8 connection(s)
I20260923 23:05:36.768496 10225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:05:36.772064 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.769390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:41507 (local address 127.5.74.62:34041)
0923 23:05:36.769487 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.769490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.769701 (+ 211us) server_negotiation.cc:407] Connection header received
0923 23:05:36.770204 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.770205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.770211 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.770280 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.771712 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.771713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.771713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:36.771718 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.771737 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:36.771742 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.771816 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.771973 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.771990 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 23:05:36.772430 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.769348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34041 (local address 127.5.74.2:41507)
0923 23:05:36.769536 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.769583 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.770071 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.770404 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.770408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.770418 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:36.771516 (+ 1098us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:05:36.771516 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:36.771548 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:05:36.771582 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:36.771858 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:36.771864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:36.771867 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.771950 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.772007 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260923 23:05:36.773049 10353 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34041
I20260923 23:05:36.773147 10353 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:36.773275 10353 heartbeater.cc:511] Master 127.5.74.62:34041 requested a full tablet report, sending...
I20260923 23:05:36.773677 10034 ts_manager.cc:194] Registered new tserver with Master: 1045643f73404477bc4f069d7abed9a0 (127.5.74.2:32933)
I20260923 23:05:36.777837 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 10225
I20260923 23:05:36.777952 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:36.778110 5416 external_mini_cluster.cc:1428] Setting key d9dfd8927a3078a00fbf1c6060ffb190
I20260923 23:05:36.779400 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34041
--builtin_ntp_servers=127.5.74.20:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 23:05:36.862534 10356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:36.862751 10356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:36.862775 10356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:36.863986 10356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:36.864056 10356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:36.864101 10356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:36.865514 10356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:36.865788 10356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:36.866088 10356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:36.871294 10364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.871294 10362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:36.871532 10361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:36.871628 10356 server_base.cc:1034] running on GCE node
I20260923 23:05:36.871974 10356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:36.872401 10356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:36.873581 10356 hybrid_clock.cc:648] HybridClock initialized: now 1790204736873566 us; error 29 us; skew 500 ppm
I20260923 23:05:36.875003 10356 webserver.cc:460] Webserver started at http://127.5.74.3:41805/ using document root <none> and password file <none>
I20260923 23:05:36.875270 10356 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:36.875322 10356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:36.875481 10356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:36.876829 10356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "54bc70358d2f4734a18958f8e76523ca"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "eeeeeb4ed9921cd1382889295e423a4e"
server_key_iv: "c806ff8fdf8bcb697249b2bc857deaf7"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.877413 10356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "54bc70358d2f4734a18958f8e76523ca"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "eeeeeb4ed9921cd1382889295e423a4e"
server_key_iv: "c806ff8fdf8bcb697249b2bc857deaf7"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.878937 10356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:36.879901 10370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.880216 10356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:36.880321 10356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "54bc70358d2f4734a18958f8e76523ca"
format_stamp: "Formatted at 2026-09-23 23:05:36 on dist-test-slave-j4c5"
server_key: "eeeeeb4ed9921cd1382889295e423a4e"
server_key_iv: "c806ff8fdf8bcb697249b2bc857deaf7"
server_key_version: "encryptionkey@0"
I20260923 23:05:36.880404 10356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:36.883975 10356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:36.884306 10356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:36.884454 10356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:36.884685 10356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:36.885016 10356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:36.885049 10356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.885072 10356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:36.885084 10356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:36.891182 10356 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:40231
I20260923 23:05:36.891223 10483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:40231 every 8 connection(s)
I20260923 23:05:36.891597 10356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:05:36.894594 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 10356
I20260923 23:05:36.894685 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:36.894941 5416 external_mini_cluster.cc:1428] Setting key c4c4c164f3b836fb1202a30374681064
I20260923 23:05:36.895090 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.892520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:35325 (local address 127.5.74.62:34041)
0923 23:05:36.892607 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.892609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.892815 (+ 206us) server_negotiation.cc:407] Connection header received
0923 23:05:36.893401 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.893403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.893411 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.893508 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.894785 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.894788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.894788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:36.894793 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.894813 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:36.894818 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.894913 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.894994 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.895014 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 23:05:36.895305 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.892469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34041 (local address 127.5.74.3:35325)
0923 23:05:36.892682 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.892718 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.893219 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.893570 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.893573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.893582 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:36.894560 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:05:36.894560 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:36.894599 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:05:36.894630 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:36.894910 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:36.894914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:36.894916 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.894978 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.895058 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260923 23:05:36.896026 10484 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34041
I20260923 23:05:36.896174 10484 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:36.896353 10484 heartbeater.cc:511] Master 127.5.74.62:34041 requested a full tablet report, sending...
I20260923 23:05:36.896842 10033 ts_manager.cc:194] Registered new tserver with Master: 54bc70358d2f4734a18958f8e76523ca (127.5.74.3:40231)
I20260923 23:05:36.907529 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:05:36.910940 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.909140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.5.74.62:34041)
0923 23:05:36.909209 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.909211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.909555 (+ 344us) server_negotiation.cc:407] Connection header received
0923 23:05:36.909592 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.909593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.909598 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.909658 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.910551 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.910553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.910553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:36.910557 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.910572 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:36.910576 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.910648 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.910816 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.910835 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:05:36.913614 10033 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:36.915311 10033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:36.919757 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.918013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:45899 (local address 127.0.0.1:59872)
0923 23:05:36.918155 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.918197 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.918265 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.918838 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.918839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.918843 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:36.919272 (+ 429us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:05:36.919272 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:36.919300 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:05:36.919329 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:36.919571 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:36.919578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:36.919582 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.919655 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.919670 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260923 23:05:36.919792 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.918307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.5.74.1:45899)
0923 23:05:36.918689 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.918693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.918701 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:36.918725 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.918729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.918736 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.919020 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.919419 (+ 399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.919427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.919427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:36.919432 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.919467 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:36.919493 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.919566 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.919665 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.919681 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":78,"threads_started":1}
I20260923 23:05:36.919941 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.918019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:40231 (local address 127.0.0.1:51080)
0923 23:05:36.918360 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.918368 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.918412 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.919136 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.919137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.919140 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:36.919552 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:05:36.919553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:36.919561 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:05:36.919573 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:36.919796 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:36.919796 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:36.919797 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.919839 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.919852 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":4,"thread_start_us":97,"threads_started":1}
I20260923 23:05:36.920189 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.918019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:32933 (local address 127.0.0.1:50374)
0923 23:05:36.918237 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:36.918246 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:36.918280 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:36.918611 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:36.918616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:36.918624 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:36.919760 (+ 1136us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:05:36.919761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:36.919768 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:05:36.919791 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:36.920028 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:36.920029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:36.920029 (+ 0us) client_negotiation.cc:769] Sending connection context
0923 23:05:36.920090 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:05:36.920105 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":101,"threads_started":1}
I20260923 23:05:36.920231 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.918286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.5.74.2:32933)
0923 23:05:36.918451 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.918457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.918467 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:36.918493 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.918495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.918505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.918607 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.919883 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.919892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.919892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:36.919897 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.919928 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:36.919950 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.920033 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.920098 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.920113 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260923 23:05:36.920228 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:36.918485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.5.74.3:40231)
0923 23:05:36.918648 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:36.918653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:36.918664 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:36.918692 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:36.918695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:36.918703 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:36.918789 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:36.919660 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:36.919667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:36.919667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:36.919671 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:36.919701 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:36.919722 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:36.919772 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:36.920095 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 23:05:36.920114 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260923 23:05:36.920746 10287 tablet_service.cc:1467] Processing CreateTablet for tablet d67fba03d4034889b43166cd83b0fb0e (DEFAULT_TABLE table=test-table [id=63fcb154221448118d932f230a2aa601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:36.920748 10156 tablet_service.cc:1467] Processing CreateTablet for tablet d67fba03d4034889b43166cd83b0fb0e (DEFAULT_TABLE table=test-table [id=63fcb154221448118d932f230a2aa601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:36.920954 10418 tablet_service.cc:1467] Processing CreateTablet for tablet d67fba03d4034889b43166cd83b0fb0e (DEFAULT_TABLE table=test-table [id=63fcb154221448118d932f230a2aa601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:36.921113 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67fba03d4034889b43166cd83b0fb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:36.921113 10156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67fba03d4034889b43166cd83b0fb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:36.921314 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67fba03d4034889b43166cd83b0fb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:36.925038 10500 tablet_bootstrap.cc:492] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: Bootstrap starting.
I20260923 23:05:36.925691 10500 tablet_bootstrap.cc:654] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:36.926015 10500 log.cc:824] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:36.926349 10499 tablet_bootstrap.cc:492] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1: Bootstrap starting.
I20260923 23:05:36.927043 10499 tablet_bootstrap.cc:654] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:36.927040 10500 tablet_bootstrap.cc:492] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: No bootstrap required, opened a new log
I20260923 23:05:36.927167 10500 ts_tablet_manager.cc:1397] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:05:36.927369 10499 log.cc:824] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:36.927733 10498 tablet_bootstrap.cc:492] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: Bootstrap starting.
I20260923 23:05:36.928411 10499 tablet_bootstrap.cc:492] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1: No bootstrap required, opened a new log
I20260923 23:05:36.928537 10499 ts_tablet_manager.cc:1397] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:36.928592 10498 tablet_bootstrap.cc:654] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:36.928908 10498 log.cc:824] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:36.929999 10498 tablet_bootstrap.cc:492] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: No bootstrap required, opened a new log
I20260923 23:05:36.930087 10498 ts_tablet_manager.cc:1397] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:05:36.930459 10500 raft_consensus.cc:348] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } }
I20260923 23:05:36.930644 10500 raft_consensus.cc:374] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:36.930677 10500 raft_consensus.cc:729] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1045643f73404477bc4f069d7abed9a0, State: Initialized, Role: FOLLOWER
I20260923 23:05:36.930825 10500 consensus_queue.cc:260] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } }
I20260923 23:05:36.931108 10500 ts_tablet_manager.cc:1428] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 23:05:36.931238 10353 heartbeater.cc:503] Master 127.5.74.62:34041 was elected leader, sending a full tablet report...
I20260923 23:05:36.931789 10499 raft_consensus.cc:348] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } }
I20260923 23:05:36.932019 10499 raft_consensus.cc:374] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:36.932061 10499 raft_consensus.cc:729] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a565ccc8ba4edc9778785d7c484dd1, State: Initialized, Role: FOLLOWER
I20260923 23:05:36.932216 10499 consensus_queue.cc:260] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } }
I20260923 23:05:36.932559 10499 ts_tablet_manager.cc:1428] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260923 23:05:36.932571 10222 heartbeater.cc:503] Master 127.5.74.62:34041 was elected leader, sending a full tablet report...
I20260923 23:05:36.932813 10498 raft_consensus.cc:348] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } }
I20260923 23:05:36.932986 10498 raft_consensus.cc:374] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:36.933025 10498 raft_consensus.cc:729] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bc70358d2f4734a18958f8e76523ca, State: Initialized, Role: FOLLOWER
I20260923 23:05:36.933156 10498 consensus_queue.cc:260] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } }
I20260923 23:05:36.933466 10484 heartbeater.cc:503] Master 127.5.74.62:34041 was elected leader, sending a full tablet report...
I20260923 23:05:36.933466 10498 ts_tablet_manager.cc:1428] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
W20260923 23:05:37.019140 10354 tablet.cc:2369] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:37.061697 10504 raft_consensus.cc:482] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:37.061843 10504 raft_consensus.cc:504] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } }
I20260923 23:05:37.062237 10504 leader_election.cc:288] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54bc70358d2f4734a18958f8e76523ca (127.5.74.3:40231), 43a565ccc8ba4edc9778785d7c484dd1 (127.5.74.1:45899)
I20260923 23:05:37.070742 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.062561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:34979 (local address 127.5.74.1:45899)
0923 23:05:37.062617 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:37.062619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:37.062766 (+ 147us) server_negotiation.cc:407] Connection header received
0923 23:05:37.062813 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:37.062813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:37.062817 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:37.062858 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:37.070379 (+ 7521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:37.070381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:37.070381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:37.070387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:37.070409 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:37.070415 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:37.070498 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:37.070638 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 23:05:37.070652 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:05:37.070742 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.062502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:33131 (local address 127.5.74.3:40231)
0923 23:05:37.062546 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:37.062548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:37.062559 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:37.062654 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:37.062655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:37.062661 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:37.062712 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:37.070379 (+ 7667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:37.070381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:37.070381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:37.070387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:37.070409 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:37.070415 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:37.070498 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:37.070635 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 23:05:37.070652 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:37.070802 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.062444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:45899 (local address 127.5.74.2:34979)
0923 23:05:37.062652 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:37.062662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:37.062749 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:37.062882 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:37.062883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:37.062885 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:37.070146 (+ 7261us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:05:37.070147 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:37.070161 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:05:37.070182 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:37.070544 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:37.070546 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:37.070547 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:05:37.070629 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 23:05:37.070671 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":83,"thread_start_us":99,"threads_started":1}
I20260923 23:05:37.070802 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.062436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:40231 (local address 127.5.74.2:33131)
0923 23:05:37.062504 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:37.062516 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:37.062563 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:37.062749 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:37.062751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:37.062753 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:37.070146 (+ 7393us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:05:37.070147 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:37.070158 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:05:37.070181 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:37.070544 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:37.070546 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:37.070547 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:05:37.070627 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 23:05:37.070661 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 23:05:37.071237 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67fba03d4034889b43166cd83b0fb0e" candidate_uuid: "1045643f73404477bc4f069d7abed9a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bc70358d2f4734a18958f8e76523ca" is_pre_election: true
I20260923 23:05:37.071224 10176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67fba03d4034889b43166cd83b0fb0e" candidate_uuid: "1045643f73404477bc4f069d7abed9a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a565ccc8ba4edc9778785d7c484dd1" is_pre_election: true
I20260923 23:05:37.071409 10438 raft_consensus.cc:2454] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1045643f73404477bc4f069d7abed9a0 in term 0.
I20260923 23:05:37.071409 10176 raft_consensus.cc:2454] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1045643f73404477bc4f069d7abed9a0 in term 0.
I20260923 23:05:37.071668 10243 leader_election.cc:302] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1045643f73404477bc4f069d7abed9a0, 43a565ccc8ba4edc9778785d7c484dd1; no voters:
I20260923 23:05:37.071810 10504 raft_consensus.cc:2781] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:37.071866 10504 raft_consensus.cc:482] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:37.071900 10504 raft_consensus.cc:3037] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:37.072639 10504 raft_consensus.cc:504] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } }
I20260923 23:05:37.072799 10504 leader_election.cc:288] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [CANDIDATE]: Term 1 election: Requested vote from peers 54bc70358d2f4734a18958f8e76523ca (127.5.74.3:40231), 43a565ccc8ba4edc9778785d7c484dd1 (127.5.74.1:45899)
I20260923 23:05:37.072973 10176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67fba03d4034889b43166cd83b0fb0e" candidate_uuid: "1045643f73404477bc4f069d7abed9a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a565ccc8ba4edc9778785d7c484dd1"
I20260923 23:05:37.073056 10176 raft_consensus.cc:3037] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:37.073076 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67fba03d4034889b43166cd83b0fb0e" candidate_uuid: "1045643f73404477bc4f069d7abed9a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bc70358d2f4734a18958f8e76523ca"
I20260923 23:05:37.073176 10438 raft_consensus.cc:3037] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:37.073822 10176 raft_consensus.cc:2454] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1045643f73404477bc4f069d7abed9a0 in term 1.
I20260923 23:05:37.073985 10243 leader_election.cc:302] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1045643f73404477bc4f069d7abed9a0, 43a565ccc8ba4edc9778785d7c484dd1; no voters:
I20260923 23:05:37.074016 10438 raft_consensus.cc:2454] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1045643f73404477bc4f069d7abed9a0 in term 1.
I20260923 23:05:37.074093 10504 raft_consensus.cc:2781] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:37.074263 10504 raft_consensus.cc:686] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 1 LEADER]: Becoming Leader. State: Replica: 1045643f73404477bc4f069d7abed9a0, State: Running, Role: LEADER
I20260923 23:05:37.074383 10504 consensus_queue.cc:237] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } }
I20260923 23:05:37.075230 10033 catalog_manager.cc:5495] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1045643f73404477bc4f069d7abed9a0 (127.5.74.2). New cstate: current_term: 1 leader_uuid: "1045643f73404477bc4f069d7abed9a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1045643f73404477bc4f069d7abed9a0" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 32933 } health_report { overall_health: HEALTHY } } }
I20260923 23:05:37.108856 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.107610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.5.74.2:32933)
0923 23:05:37.107662 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:37.107664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:37.107674 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:37.107715 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:37.107716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:37.107732 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:37.107815 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:37.108482 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:37.108483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:37.108484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:37.108489 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:37.108569 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:37.108580 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:37.108690 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:37.108759 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 23:05:37.108773 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:05:37.111155 10438 raft_consensus.cc:1261] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [term 1 FOLLOWER]: Refusing update from remote peer 1045643f73404477bc4f069d7abed9a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:37.111187 10176 raft_consensus.cc:1261] T d67fba03d4034889b43166cd83b0fb0e P 43a565ccc8ba4edc9778785d7c484dd1 [term 1 FOLLOWER]: Refusing update from remote peer 1045643f73404477bc4f069d7abed9a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:37.111557 10504 consensus_queue.cc:1035] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a565ccc8ba4edc9778785d7c484dd1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 45899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:37.111738 10508 consensus_queue.cc:1035] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54bc70358d2f4734a18958f8e76523ca" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:37.114202 10511 mvcc.cc:205] Tried to move back new op lower bound from 7332678603204284416 to 7332678603057250304. Current Snapshot: MvccSnapshot[applied={T|T < 7332678603204284416}]
I20260923 23:05:37.114523 10513 mvcc.cc:205] Tried to move back new op lower bound from 7332678603204284416 to 7332678603057250304. Current Snapshot: MvccSnapshot[applied={T|T < 7332678603204284416}]
I20260923 23:05:37.115381 10512 mvcc.cc:205] Tried to move back new op lower bound from 7332678603204284416 to 7332678603057250304. Current Snapshot: MvccSnapshot[applied={T|T < 7332678603204284416 or (T in {7332678603204284416})}]
I20260923 23:05:37.117794 10033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:37.117921 10033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:37.119125 10033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec624570228745a78b8e276d75c351bc: Sending DeleteTablet for 3 replicas of tablet d67fba03d4034889b43166cd83b0fb0e
I20260923 23:05:37.119587 10287 tablet_service.cc:1514] Processing DeleteTablet for tablet d67fba03d4034889b43166cd83b0fb0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:37 UTC) from {username='slave'} at 127.0.0.1:50374
I20260923 23:05:37.119603 10418 tablet_service.cc:1514] Processing DeleteTablet for tablet d67fba03d4034889b43166cd83b0fb0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:37 UTC) from {username='slave'} at 127.0.0.1:51080
I20260923 23:05:37.119968 10156 tablet_service.cc:1514] Processing DeleteTablet for tablet d67fba03d4034889b43166cd83b0fb0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:37 UTC) from {username='slave'} at 127.0.0.1:59872
I20260923 23:05:37.120066 10522 tablet_replica.cc:330] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: stopping tablet replica
I20260923 23:05:37.120237 10522 raft_consensus.cc:2229] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:37.120389 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 10094
I20260923 23:05:37.120462 10522 raft_consensus.cc:2258] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:37.124253 10521 tablet_replica.cc:330] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: stopping tablet replica
I20260923 23:05:37.124406 10521 raft_consensus.cc:2229] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:37.124588 10521 raft_consensus.cc:2258] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:37.125208 10521 ts_tablet_manager.cc:1905] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:37.126080 10522 ts_tablet_manager.cc:1905] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:37.127061 10521 ts_tablet_manager.cc:1918] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:37.127147 10521 log.cc:1196] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/d67fba03d4034889b43166cd83b0fb0e
I20260923 23:05:37.127447 10521 ts_tablet_manager.cc:1939] T d67fba03d4034889b43166cd83b0fb0e P 1045643f73404477bc4f069d7abed9a0: Deleting consensus metadata
I20260923 23:05:37.127930 10019 catalog_manager.cc:4841] TS 1045643f73404477bc4f069d7abed9a0 (127.5.74.2:32933): tablet d67fba03d4034889b43166cd83b0fb0e (table test-table [id=63fcb154221448118d932f230a2aa601]) successfully deleted
I20260923 23:05:37.129107 10522 ts_tablet_manager.cc:1918] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:37.133029 10522 log.cc:1196] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/d67fba03d4034889b43166cd83b0fb0e
W20260923 23:05:37.133039 10021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:45899 (error 108)
I20260923 23:05:37.133380 10522 ts_tablet_manager.cc:1939] T d67fba03d4034889b43166cd83b0fb0e P 54bc70358d2f4734a18958f8e76523ca: Deleting consensus metadata
I20260923 23:05:37.133878 10018 catalog_manager.cc:4841] TS 54bc70358d2f4734a18958f8e76523ca (127.5.74.3:40231): tablet d67fba03d4034889b43166cd83b0fb0e (table test-table [id=63fcb154221448118d932f230a2aa601]) successfully deleted
I20260923 23:05:37.133912 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 10225
I20260923 23:05:37.134887 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.134727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:45899 (local address 127.0.0.1:59888)
0923 23:05:37.134797 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:37.134850 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 23:05:37.135048 10021 catalog_manager.cc:4584] TS 43a565ccc8ba4edc9778785d7c484dd1 (127.5.74.1:45899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d67fba03d4034889b43166cd83b0fb0e: Network error: Client connection negotiation failed: client connection to 127.5.74.1:45899: connect: Connection refused (error 111)
I20260923 23:05:37.139297 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 10356
I20260923 23:05:37.144335 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 10003
2026-09-23T23:05:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (815 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 23:05:37.165045 5416 mini_oidc.cc:152] Starting JWKS server
I20260923 23:05:37.166363 5416 webserver.cc:460] Webserver started at https://127.0.0.1:33367/ using document root <none> and password file <none>
I20260923 23:05:37.166472 5416 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 23:05:37.166630 5416 webserver.cc:460] Webserver started at http://127.0.0.1:43443/ using document root <none> and password file <none>
2026-09-23T23:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:37Z Disabled control of system clock
I20260923 23:05:37.175900 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41449
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:41449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33367/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/testchainca.cert with env {}
W20260923 23:05:37.264345 10530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:37.264569 10530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:37.264593 10530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:37.265868 10530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:37.265920 10530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:37.265934 10530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:37.265954 10530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:37.265966 10530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 23:05:37.266023 10530 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 23:05:37.266039 10530 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33367/jwks.json
I20260923 23:05:37.267675 10530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:41449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33367/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:37.267966 10530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:37.268313 10530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:37.273627 10538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:37.273811 10536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:37.273658 10535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:37.273931 10530 server_base.cc:1034] running on GCE node
I20260923 23:05:37.274212 10530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:37.274506 10530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:37.275695 10530 hybrid_clock.cc:648] HybridClock initialized: now 1790204737275681 us; error 32 us; skew 500 ppm
I20260923 23:05:37.277431 10530 webserver.cc:460] Webserver started at http://127.5.74.62:45021/ using document root <none> and password file <none>
I20260923 23:05:37.277730 10530 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:37.277812 10530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:37.278112 10530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:37.279440 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "b318ee4131b64e54b2a5cff00bfde063"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "3cfc7c4e4986f6853e3358382ac5bf4c"
server_key_iv: "547a872b9b367db474ba8ff816d2729a"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.279840 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "b318ee4131b64e54b2a5cff00bfde063"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "3cfc7c4e4986f6853e3358382ac5bf4c"
server_key_iv: "547a872b9b367db474ba8ff816d2729a"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.282079 10530 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260923 23:05:37.283200 10544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:37.283609 10530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:37.283737 10530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "b318ee4131b64e54b2a5cff00bfde063"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "3cfc7c4e4986f6853e3358382ac5bf4c"
server_key_iv: "547a872b9b367db474ba8ff816d2729a"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.283963 10530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:37.298740 10530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:37.299109 10530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:37.299271 10530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:37.303656 10530 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:41449
I20260923 23:05:37.303694 10596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:41449 every 8 connection(s)
I20260923 23:05:37.304062 10530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:37.305207 10597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:37.307634 10597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063: Bootstrap starting.
I20260923 23:05:37.308221 10597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:37.308475 10597 log.cc:824] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:37.309197 10597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063: No bootstrap required, opened a new log
I20260923 23:05:37.310797 10597 raft_consensus.cc:348] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b318ee4131b64e54b2a5cff00bfde063" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41449 } }
I20260923 23:05:37.310945 10597 raft_consensus.cc:374] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:37.310968 10597 raft_consensus.cc:729] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b318ee4131b64e54b2a5cff00bfde063, State: Initialized, Role: FOLLOWER
I20260923 23:05:37.311128 10597 consensus_queue.cc:260] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b318ee4131b64e54b2a5cff00bfde063" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41449 } }
I20260923 23:05:37.311223 10597 raft_consensus.cc:388] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:37.311259 10597 raft_consensus.cc:482] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:37.311309 10597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:37.311396 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 10530
I20260923 23:05:37.311504 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:37.311695 5416 external_mini_cluster.cc:1428] Setting key 16d6566463acdcaf1419721200ef9566
I20260923 23:05:37.312099 10597 raft_consensus.cc:504] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b318ee4131b64e54b2a5cff00bfde063" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41449 } }
I20260923 23:05:37.312243 10597 leader_election.cc:302] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b318ee4131b64e54b2a5cff00bfde063; no voters:
I20260923 23:05:37.312439 10597 leader_election.cc:288] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:37.312529 10601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:37.312668 10601 raft_consensus.cc:686] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [term 1 LEADER]: Becoming Leader. State: Replica: b318ee4131b64e54b2a5cff00bfde063, State: Running, Role: LEADER
I20260923 23:05:37.312726 10597 sys_catalog.cc:564] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:37.312755 10601 consensus_queue.cc:237] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b318ee4131b64e54b2a5cff00bfde063" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41449 } }
I20260923 23:05:37.313129 10604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b318ee4131b64e54b2a5cff00bfde063. Latest consensus state: current_term: 1 leader_uuid: "b318ee4131b64e54b2a5cff00bfde063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b318ee4131b64e54b2a5cff00bfde063" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41449 } } }
I20260923 23:05:37.313215 10603 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b318ee4131b64e54b2a5cff00bfde063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b318ee4131b64e54b2a5cff00bfde063" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41449 } } }
I20260923 23:05:37.313293 10603 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:37.313226 10604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:37.313483 10610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:37.313999 10610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:37.315538 10610 catalog_manager.cc:1347] Generated new cluster ID: d8c88232379f45e8805aa9d3a3b47574
I20260923 23:05:37.315598 10610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:37.319062 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.312305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.5.74.62:41449)
0923 23:05:37.312674 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:37.312684 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:37.312803 (+ 119us) server_negotiation.cc:407] Connection header received
0923 23:05:37.313017 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:37.313025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:37.313109 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:37.313883 (+ 774us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:37.313999 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.315518 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.316769 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.317400 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.317531 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:37.318294 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:37.318298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:37.318299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:37.318522 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:37.318565 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:37.318582 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:37.318654 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:37.318729 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 23:05:37.318822 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260923 23:05:37.320297 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:05:37.325883 10610 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:37.326464 10610 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:37.332248 10610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b318ee4131b64e54b2a5cff00bfde063: Generated new TSK 0
I20260923 23:05:37.332561 10610 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 23:05:37.357647 10627 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 23:05:37.357788 10627 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 23:05:37.358003 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.340927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.5.74.62:41449)
0923 23:05:37.340994 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:37.340996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:37.341123 (+ 127us) server_negotiation.cc:407] Connection header received
0923 23:05:37.341204 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:37.341205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:37.341229 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:37.341318 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0923 23:05:37.341629 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.342447 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.343289 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.343606 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.343678 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:37.343905 (+ 227us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 23:05:37.357748 (+ 13843us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 23:05:37.357926 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60318: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260923 23:05:37.358067 10602 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60318: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 23:05:37.358608 10626 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:37.340820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41449 (local address 127.0.0.1:60318)
0923 23:05:37.341045 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:37.341055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:37.341117 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:37.341367 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:37.341368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:37.341370 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 23:05:37.341528 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:37.341532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.342540 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.342541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:37.343192 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:37.343194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.343686 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.343690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:37.343742 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:37.343766 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 23:05:37.358025 (+ 14259us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 23:05:37.358297 (+ 272us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.74.62:41449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":112,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.74.62:41449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 23:05:37.359107 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 10530
2026-09-23T23:05:37Z chronyd exiting
I20260923 23:05:37.396412 5416 test_util.cc:175] -----------------------------------------------
I20260923 23:05:37.396500 5416 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204714633713-5416-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 23:05:37.397305 5416 mini_oidc.cc:152] Starting JWKS server
I20260923 23:05:37.398257 5416 webserver.cc:460] Webserver started at https://127.0.0.1:36995/ using document root <none> and password file <none>
I20260923 23:05:37.398307 5416 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 23:05:37.398509 5416 webserver.cc:460] Webserver started at http://127.0.0.1:43757/ using document root <none> and password file <none>
2026-09-23T23:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:37Z Disabled control of system clock
I20260923 23:05:37.407807 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:36899
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:36899
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36995/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/testchainca.cert with env {}
W20260923 23:05:37.499979 10633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:37.500319 10633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:37.500362 10633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:37.501664 10633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:37.501750 10633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:37.501775 10633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:37.501806 10633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:37.501834 10633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 23:05:37.501857 10633 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 23:05:37.501881 10633 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36995/jwks.json
I20260923 23:05:37.503569 10633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:36899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:36899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36995/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:37.503885 10633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:37.504127 10633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:37.509867 10638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:37.509862 10639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:37.509862 10641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:37.509882 10633 server_base.cc:1034] running on GCE node
I20260923 23:05:37.510360 10633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:37.510692 10633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:37.511937 10633 hybrid_clock.cc:648] HybridClock initialized: now 1790204737511910 us; error 45 us; skew 500 ppm
I20260923 23:05:37.513592 10633 webserver.cc:460] Webserver started at http://127.5.74.62:34305/ using document root <none> and password file <none>
I20260923 23:05:37.513885 10633 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:37.513968 10633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:37.514134 10633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:37.515573 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "d414708c336d4ce5833761a9cf433e84"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "e599ed139af9e11576d10d2ee1e31077"
server_key_iv: "4bf9094dc361c79e1493f5863a8ca9fa"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.515995 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "d414708c336d4ce5833761a9cf433e84"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "e599ed139af9e11576d10d2ee1e31077"
server_key_iv: "4bf9094dc361c79e1493f5863a8ca9fa"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.517640 10633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:37.518532 10647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:37.518776 10633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:37.518898 10633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "d414708c336d4ce5833761a9cf433e84"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "e599ed139af9e11576d10d2ee1e31077"
server_key_iv: "4bf9094dc361c79e1493f5863a8ca9fa"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.519007 10633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:37.542600 10633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:37.542980 10633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:37.543157 10633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:37.547166 10633 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:36899
I20260923 23:05:37.547237 10699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:36899 every 8 connection(s)
I20260923 23:05:37.547628 10633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:37.548774 10700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:37.551332 10700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84: Bootstrap starting.
I20260923 23:05:37.551946 10700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:37.552212 10700 log.cc:824] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:37.553156 10700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84: No bootstrap required, opened a new log
I20260923 23:05:37.553930 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 10633
I20260923 23:05:37.554109 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:37.554382 5416 external_mini_cluster.cc:1428] Setting key cfb3c739b0d3cb3f5cfb2704cbc93a5d
I20260923 23:05:37.555233 10700 raft_consensus.cc:348] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414708c336d4ce5833761a9cf433e84" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36899 } }
I20260923 23:05:37.555429 10700 raft_consensus.cc:374] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:37.555477 10700 raft_consensus.cc:729] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d414708c336d4ce5833761a9cf433e84, State: Initialized, Role: FOLLOWER
I20260923 23:05:37.555658 10700 consensus_queue.cc:260] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414708c336d4ce5833761a9cf433e84" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36899 } }
I20260923 23:05:37.555771 10700 raft_consensus.cc:388] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:37.555807 10700 raft_consensus.cc:482] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:37.555863 10700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:37.556730 10700 raft_consensus.cc:504] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414708c336d4ce5833761a9cf433e84" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36899 } }
I20260923 23:05:37.556833 10700 leader_election.cc:302] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d414708c336d4ce5833761a9cf433e84; no voters:
I20260923 23:05:37.557070 10700 leader_election.cc:288] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:37.557099 10705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:37.557221 10705 raft_consensus.cc:686] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [term 1 LEADER]: Becoming Leader. State: Replica: d414708c336d4ce5833761a9cf433e84, State: Running, Role: LEADER
I20260923 23:05:37.557307 10705 consensus_queue.cc:237] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414708c336d4ce5833761a9cf433e84" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36899 } }
I20260923 23:05:37.557379 10700 sys_catalog.cc:564] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:37.557685 10707 sys_catalog.cc:455] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d414708c336d4ce5833761a9cf433e84. Latest consensus state: current_term: 1 leader_uuid: "d414708c336d4ce5833761a9cf433e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414708c336d4ce5833761a9cf433e84" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36899 } } }
I20260923 23:05:37.557695 10706 sys_catalog.cc:455] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d414708c336d4ce5833761a9cf433e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414708c336d4ce5833761a9cf433e84" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36899 } } }
I20260923 23:05:37.557886 10706 sys_catalog.cc:458] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:37.557852 10707 sys_catalog.cc:458] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84 [sys.catalog]: This master's current role is: LEADER
W20260923 23:05:37.559674 10721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:05:37.559723 10721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:05:37.559785 10711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:37.559964 10711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:37.561637 10711 catalog_manager.cc:1347] Generated new cluster ID: 38dc59d2a1234c3bae06cbf301c43c41
I20260923 23:05:37.561723 10711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:37.561901 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.555000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.5.74.62:36899)
0923 23:05:37.555323 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:37.555334 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:37.555367 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:05:37.555487 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:37.555495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:37.555589 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:37.555742 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:37.555987 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.557858 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.558973 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.559860 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.560033 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:37.560933 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:37.560937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:37.560938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:37.561193 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:37.561251 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:37.561269 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:37.561368 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:37.561455 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 23:05:37.561543 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":187,"threads_started":1}
I20260923 23:05:37.562914 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:05:37.570755 10711 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:37.571333 10711 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:37.579779 10711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d414708c336d4ce5833761a9cf433e84: Generated new TSK 0
I20260923 23:05:37.580049 10711 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 23:05:37.604135 10730 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 23:05:37.604436 10730 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260923 23:05:37.604735 10729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:37.588265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36899 (local address 127.0.0.1:47234)
0923 23:05:37.588651 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:37.588665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:37.588737 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:37.588918 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:37.588919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:37.588920 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0923 23:05:37.588982 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:37.588984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.590090 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.590092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:37.590838 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:37.590841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.591380 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.591385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:37.591456 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:37.591463 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 23:05:37.604521 (+ 13058us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 23:05:37.604628 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.74.62:36899: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":175,"threads_started":1}
I20260923 23:05:37.605165 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.588417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.5.74.62:36899)
0923 23:05:37.588490 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:37.588493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:37.588740 (+ 247us) server_negotiation.cc:407] Connection header received
0923 23:05:37.588807 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:37.588809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:37.588823 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:37.588912 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0923 23:05:37.589062 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.589872 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.590952 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.591265 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.591347 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:37.591593 (+ 246us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 23:05:37.604372 (+ 12779us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 23:05:37.604979 (+ 607us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 23:05:37.605223 10704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260923 23:05:37.605306 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 10633
2026-09-23T23:05:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (246 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 23:05:37.644027 5416 mini_oidc.cc:152] Starting JWKS server
I20260923 23:05:37.644809 5416 webserver.cc:460] Webserver started at https://127.0.0.1:38233/ using document root <none> and password file <none>
I20260923 23:05:37.644887 5416 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 23:05:37.645046 5416 webserver.cc:460] Webserver started at http://127.0.0.1:45179/ using document root <none> and password file <none>
2026-09-23T23:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:37Z Disabled control of system clock
I20260923 23:05:37.654173 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:36191
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:36191
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38233/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 23:05:37.740916 10736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:37.741134 10736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:37.741158 10736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:37.742305 10736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:37.742355 10736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:37.742369 10736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:37.742381 10736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:37.742393 10736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 23:05:37.742408 10736 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 23:05:37.742419 10736 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38233/jwks.json
I20260923 23:05:37.744026 10736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:36191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:36191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38233/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:37.744374 10736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:37.744650 10736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:05:37.749838 10736 server_base.cc:1034] running on GCE node
W20260923 23:05:37.749748 10742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:37.749826 10741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:37.749748 10744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:37.750407 10736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:37.750731 10736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:37.751931 10736 hybrid_clock.cc:648] HybridClock initialized: now 1790204737751882 us; error 76 us; skew 500 ppm
I20260923 23:05:37.753484 10736 webserver.cc:460] Webserver started at http://127.5.74.62:42167/ using document root <none> and password file <none>
I20260923 23:05:37.753737 10736 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:37.753819 10736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:37.753968 10736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:37.755064 10736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "7ba790348b4c4851b39e3c152b0e2cb5"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "ee42350e7d3cd95a990e74054fd21c57"
server_key_iv: "b466c6b5901cb53d00815e2e76468b9e"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.755434 10736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "7ba790348b4c4851b39e3c152b0e2cb5"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "ee42350e7d3cd95a990e74054fd21c57"
server_key_iv: "b466c6b5901cb53d00815e2e76468b9e"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.756994 10736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:37.757813 10750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:37.758116 10736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:37.758226 10736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "7ba790348b4c4851b39e3c152b0e2cb5"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "ee42350e7d3cd95a990e74054fd21c57"
server_key_iv: "b466c6b5901cb53d00815e2e76468b9e"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.758309 10736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:37.769091 10736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:37.769475 10736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:37.769649 10736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:37.774509 10736 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:36191
I20260923 23:05:37.774641 10802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:36191 every 8 connection(s)
I20260923 23:05:37.774956 10736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:37.775848 10803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:37.778713 10803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5: Bootstrap starting.
I20260923 23:05:37.779393 10803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:37.779572 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 10736
I20260923 23:05:37.779681 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:37.779788 10803 log.cc:824] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:37.779886 5416 external_mini_cluster.cc:1428] Setting key c4681f245716f370b3245e2f65f8367d
I20260923 23:05:37.780742 10803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5: No bootstrap required, opened a new log
I20260923 23:05:37.783327 10803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba790348b4c4851b39e3c152b0e2cb5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36191 } }
I20260923 23:05:37.783545 10803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:37.783576 10803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba790348b4c4851b39e3c152b0e2cb5, State: Initialized, Role: FOLLOWER
I20260923 23:05:37.783733 10803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba790348b4c4851b39e3c152b0e2cb5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36191 } }
I20260923 23:05:37.783820 10803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:37.783860 10803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:37.783910 10803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:37.784689 10803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba790348b4c4851b39e3c152b0e2cb5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36191 } }
I20260923 23:05:37.784816 10803 leader_election.cc:302] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ba790348b4c4851b39e3c152b0e2cb5; no voters:
I20260923 23:05:37.785017 10803 leader_election.cc:288] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:37.785207 10809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:37.785326 10803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:37.785513 10809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [term 1 LEADER]: Becoming Leader. State: Replica: 7ba790348b4c4851b39e3c152b0e2cb5, State: Running, Role: LEADER
I20260923 23:05:37.785652 10809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba790348b4c4851b39e3c152b0e2cb5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36191 } }
I20260923 23:05:37.786088 10808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ba790348b4c4851b39e3c152b0e2cb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba790348b4c4851b39e3c152b0e2cb5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36191 } } }
I20260923 23:05:37.786145 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.780496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.5.74.62:36191)
0923 23:05:37.780703 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:37.780711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:37.780737 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:05:37.781254 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:37.781261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:37.781334 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:37.781453 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:37.781771 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.782728 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.783900 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.784434 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.784538 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:37.785375 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:37.785380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:37.785380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:37.785600 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:37.785647 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:37.785661 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:37.785703 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:37.785835 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 23:05:37.785930 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260923 23:05:37.786110 10810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ba790348b4c4851b39e3c152b0e2cb5. Latest consensus state: current_term: 1 leader_uuid: "7ba790348b4c4851b39e3c152b0e2cb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba790348b4c4851b39e3c152b0e2cb5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36191 } } }
I20260923 23:05:37.786167 10808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:37.786193 10810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:37.786499 10817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:37.786990 10817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:37.787319 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:05:37.789196 10817 catalog_manager.cc:1347] Generated new cluster ID: 1f8ed87cc4c04788a483b82d6cc99b13
I20260923 23:05:37.789285 10817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:37.803795 10817 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:37.804587 10817 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:37.811664 10817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ba790348b4c4851b39e3c152b0e2cb5: Generated new TSK 0
I20260923 23:05:37.811964 10817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:05:37.829391 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:37.825875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.5.74.62:36191)
0923 23:05:37.825965 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:37.825967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:37.825974 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:05:37.826013 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:37.826014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:37.826025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:37.826109 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0923 23:05:37.826259 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.826780 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.827671 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.827891 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.827972 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:37.828220 (+ 248us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 23:05:37.829177 (+ 957us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38233/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 23:05:37.829310 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54474: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38233/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 23:05:37.829457 10807 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54474: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38233/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 23:05:37.829524 10832 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:37.825626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36191 (local address 127.0.0.1:54474)
0923 23:05:37.825780 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:37.825791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:37.825858 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:37.826110 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:37.826111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:37.826113 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 23:05:37.826174 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:37.826176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.826933 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.826934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:37.827573 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:37.827576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:37.827969 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:37.827978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:37.828040 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:37.828046 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 23:05:37.829288 (+ 1242us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38233/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 23:05:37.829417 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.74.62:36191: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38233/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260923 23:05:37.829973 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 10736
E20260923 23:05:37.840999 10833 webserver.cc:533] Webserver: sslize failed:
E20260923 23:05:37.841111 10833 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T23:05:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (223 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T23:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:37Z Disabled control of system clock
I20260923 23:05:37.875945 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:46055
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:46055
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/testchainca.cert with env {}
W20260923 23:05:37.961658 10837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:37.961951 10837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:37.961992 10837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:37.963270 10837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:37.963354 10837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:37.963377 10837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:37.963399 10837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:37.963428 10837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 23:05:37.963454 10837 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 23:05:37.963477 10837 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 23:05:37.965236 10837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:46055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:46055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:37.965564 10837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:37.965813 10837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:37.971268 10842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:37.971297 10845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:37.971529 10843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:37.971899 10837 server_base.cc:1034] running on GCE node
I20260923 23:05:37.972224 10837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:37.972613 10837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:37.973843 10837 hybrid_clock.cc:648] HybridClock initialized: now 1790204737973828 us; error 30 us; skew 500 ppm
I20260923 23:05:37.975350 10837 webserver.cc:460] Webserver started at http://127.5.74.62:39373/ using document root <none> and password file <none>
I20260923 23:05:37.975625 10837 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:37.975701 10837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:37.975847 10837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:37.977316 10837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "7a88589cd663476cb7ba67c45512e83c"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "0c285f4a7f5b249ee3a0e3d4388fd5c3"
server_key_iv: "08b3c808ae6c81bc183b2ef47c6c2328"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.977798 10837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "7a88589cd663476cb7ba67c45512e83c"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "0c285f4a7f5b249ee3a0e3d4388fd5c3"
server_key_iv: "08b3c808ae6c81bc183b2ef47c6c2328"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.979215 10837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:05:37.980379 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:37.980765 10837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:37.980932 10837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "7a88589cd663476cb7ba67c45512e83c"
format_stamp: "Formatted at 2026-09-23 23:05:37 on dist-test-slave-j4c5"
server_key: "0c285f4a7f5b249ee3a0e3d4388fd5c3"
server_key_iv: "08b3c808ae6c81bc183b2ef47c6c2328"
server_key_version: "encryptionkey@0"
I20260923 23:05:37.981029 10837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:37.992442 10837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:37.992811 10837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:37.992985 10837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:37.997414 10837 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:46055
I20260923 23:05:37.997622 10903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:46055 every 8 connection(s)
I20260923 23:05:37.997876 10837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:37.998806 10904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:38.000770 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 10837
I20260923 23:05:38.000864 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:38.001077 5416 external_mini_cluster.cc:1428] Setting key 2602756055710eb4c98ac9fe12a5ffe9
I20260923 23:05:38.001473 10904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c: Bootstrap starting.
I20260923 23:05:38.002241 10904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:38.002600 10904 log.cc:824] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:38.003415 10904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c: No bootstrap required, opened a new log
I20260923 23:05:38.005620 10904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88589cd663476cb7ba67c45512e83c" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46055 } }
I20260923 23:05:38.005817 10904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:38.005900 10904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a88589cd663476cb7ba67c45512e83c, State: Initialized, Role: FOLLOWER
I20260923 23:05:38.006050 10904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88589cd663476cb7ba67c45512e83c" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46055 } }
I20260923 23:05:38.006145 10904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:38.006181 10904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:38.006228 10904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:38.007167 10904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88589cd663476cb7ba67c45512e83c" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46055 } }
I20260923 23:05:38.007272 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:38.001675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.5.74.62:46055)
0923 23:05:38.001921 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:38.001931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:38.001967 (+ 36us) server_negotiation.cc:407] Connection header received
0923 23:05:38.002645 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:38.002655 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:38.002739 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:38.002906 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:38.003268 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:38.004283 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:38.005089 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:38.005508 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:38.005642 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:38.006439 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:38.006443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:38.006444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:05:38.006671 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:38.006719 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:05:38.006737 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:38.006821 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:38.006924 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 23:05:38.007022 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260923 23:05:38.007285 10904 leader_election.cc:302] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a88589cd663476cb7ba67c45512e83c; no voters:
I20260923 23:05:38.007591 10904 leader_election.cc:288] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:38.007753 10909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:38.007884 10909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [term 1 LEADER]: Becoming Leader. State: Replica: 7a88589cd663476cb7ba67c45512e83c, State: Running, Role: LEADER
I20260923 23:05:38.007897 10904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:38.007980 10909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88589cd663476cb7ba67c45512e83c" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46055 } }
I20260923 23:05:38.008615 10911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a88589cd663476cb7ba67c45512e83c. Latest consensus state: current_term: 1 leader_uuid: "7a88589cd663476cb7ba67c45512e83c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88589cd663476cb7ba67c45512e83c" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46055 } } }
I20260923 23:05:38.008797 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:05:38.008806 10911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:38.009109 10916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:38.009876 10916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:38.010360 10910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a88589cd663476cb7ba67c45512e83c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88589cd663476cb7ba67c45512e83c" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46055 } } }
I20260923 23:05:38.010453 10910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:38.011722 10916 catalog_manager.cc:1347] Generated new cluster ID: 0448b5edf9894599ad1e740acb6b5268
I20260923 23:05:38.011827 10916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:38.032572 10916 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:38.033346 10916 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:38.040854 10916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a88589cd663476cb7ba67c45512e83c: Generated new TSK 0
I20260923 23:05:38.041069 10916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:05:38.102344 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:38.052244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.5.74.62:46055)
0923 23:05:38.052310 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:38.052312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:38.052424 (+ 112us) server_negotiation.cc:407] Connection header received
0923 23:05:38.052485 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:38.052486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:38.052499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:38.052589 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0923 23:05:38.052751 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:38.053532 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:38.054338 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:38.054645 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:38.054721 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:38.054852 (+ 131us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 23:05:38.101995 (+ 47143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 23:05:38.102254 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37580: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260923 23:05:38.102437 10907 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37580: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 23:05:38.102445 10933 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:38.052137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:46055 (local address 127.0.0.1:37580)
0923 23:05:38.052342 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:38.052353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:38.052423 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:38.052617 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:38.052618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:38.052621 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0923 23:05:38.052676 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:38.052678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:38.053627 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:38.053628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:38.054252 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:38.054254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:38.054720 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:38.054727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:38.054766 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:38.054771 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 23:05:38.102183 (+ 47412us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 23:05:38.102318 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.74.62:46055: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":155,"threads_started":1}
I20260923 23:05:38.103224 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 10837
2026-09-23T23:05:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (253 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (98 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (96 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:38 dist-test-slave-j4c5 krb5kdc[10945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:38 dist-test-slave-j4c5 krb5kdc[10945](info): set up 2 sockets
Sep 23 23:05:38 dist-test-slave-j4c5 krb5kdc[10945](info): commencing operation
krb5kdc: starting...
W20260923 23:05:39.376686 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:39 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:39.504001 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37721
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:37721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:39.591992 10958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:39.592330 10958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:39.592374 10958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:39.593654 10958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:39.593734 10958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:39.593757 10958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:39.593779 10958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:39.593809 10958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:39.595369 10958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:37721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:39.595669 10958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:39.595917 10958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:39.601500 10966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:39.601493 10964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:39.601500 10963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:39.601986 10958 server_base.cc:1034] running on GCE node
I20260923 23:05:39.602257 10958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:39.602593 10958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:39.603770 10958 hybrid_clock.cc:648] HybridClock initialized: now 1790204739603751 us; error 51 us; skew 500 ppm
Sep 23 23:05:39 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204739, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:39.606585 10958 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:39.607069 10958 webserver.cc:460] Webserver started at http://127.5.74.62:37439/ using document root <none> and password file <none>
I20260923 23:05:39.607331 10958 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:39.607406 10958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:39.607564 10958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:39.608776 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "09f3fb13cbfc4c568b4c066e7620f5f2"
format_stamp: "Formatted at 2026-09-23 23:05:39 on dist-test-slave-j4c5"
server_key: "28201eefce512e5966b164f9655afe66"
server_key_iv: "685fbc2884893aa55cd4bd767ae43e24"
server_key_version: "encryptionkey@0"
I20260923 23:05:39.609170 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "09f3fb13cbfc4c568b4c066e7620f5f2"
format_stamp: "Formatted at 2026-09-23 23:05:39 on dist-test-slave-j4c5"
server_key: "28201eefce512e5966b164f9655afe66"
server_key_iv: "685fbc2884893aa55cd4bd767ae43e24"
server_key_version: "encryptionkey@0"
I20260923 23:05:39.610620 10958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 23:05:39.611480 10973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:39.611749 10958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:39.611876 10958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "09f3fb13cbfc4c568b4c066e7620f5f2"
format_stamp: "Formatted at 2026-09-23 23:05:39 on dist-test-slave-j4c5"
server_key: "28201eefce512e5966b164f9655afe66"
server_key_iv: "685fbc2884893aa55cd4bd767ae43e24"
server_key_version: "encryptionkey@0"
I20260923 23:05:39.611990 10958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:39.639791 10958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:39.641969 10958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:39.642150 10958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:39.646262 10958 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:37721
I20260923 23:05:39.646404 11025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:37721 every 8 connection(s)
I20260923 23:05:39.646693 10958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:39.647639 11026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:39.649819 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 10958
I20260923 23:05:39.649946 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:39.650146 5416 external_mini_cluster.cc:1428] Setting key 020a34c5e47b04734c9b4ed34f70d44c
I20260923 23:05:39.650141 11026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2: Bootstrap starting.
I20260923 23:05:39.650869 11026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:39.651274 11026 log.cc:824] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:39.652052 11026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2: No bootstrap required, opened a new log
I20260923 23:05:39.653903 11026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3fb13cbfc4c568b4c066e7620f5f2" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37721 } }
I20260923 23:05:39.654070 11026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:39.654116 11026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f3fb13cbfc4c568b4c066e7620f5f2, State: Initialized, Role: FOLLOWER
I20260923 23:05:39.654263 11026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3fb13cbfc4c568b4c066e7620f5f2" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37721 } }
I20260923 23:05:39.654356 11026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:39.654399 11026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:39.654455 11026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:39.655078 11026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3fb13cbfc4c568b4c066e7620f5f2" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37721 } }
I20260923 23:05:39.655200 11026 leader_election.cc:302] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f3fb13cbfc4c568b4c066e7620f5f2; no voters:
I20260923 23:05:39.655393 11026 leader_election.cc:288] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:39.655421 11031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:39.655544 11031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [term 1 LEADER]: Becoming Leader. State: Replica: 09f3fb13cbfc4c568b4c066e7620f5f2, State: Running, Role: LEADER
I20260923 23:05:39.655689 11026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:39.655681 11031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3fb13cbfc4c568b4c066e7620f5f2" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37721 } }
I20260923 23:05:39.656167 11032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f3fb13cbfc4c568b4c066e7620f5f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3fb13cbfc4c568b4c066e7620f5f2" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37721 } } }
I20260923 23:05:39.656405 11032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:39.656329 11033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f3fb13cbfc4c568b4c066e7620f5f2. Latest consensus state: current_term: 1 leader_uuid: "09f3fb13cbfc4c568b4c066e7620f5f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3fb13cbfc4c568b4c066e7620f5f2" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37721 } } }
I20260923 23:05:39.656657 11033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:39.656853 11040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 23 23:05:39 dist-test-slave-j4c5 krb5kdc[10945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:39.657361 11040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:39.660615 11040 catalog_manager.cc:1347] Generated new cluster ID: deb0865fb243482883a308282934b6c1
I20260923 23:05:39.660712 11040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:39.661531 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:39.650712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.5.74.62:37721)
0923 23:05:39.650910 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:39.650919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:39.650968 (+ 49us) server_negotiation.cc:407] Connection header received
0923 23:05:39.651424 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:39.651433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:39.651493 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:39.651636 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:39.652359 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:39.653244 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:39.654003 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:39.654559 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:39.654687 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:39.657808 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:39.657812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:39.657813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:39.657825 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:39.659641 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:39.660074 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:39.660077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:39.660078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:39.660237 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:39.660564 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:39.660565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:39.660566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:39.660885 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:39.661045 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:39.661192 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 23:05:39.661324 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260923 23:05:39.689257 11040 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:39.689881 11040 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:39.694885 11040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f3fb13cbfc4c568b4c066e7620f5f2: Generated new TSK 0
I20260923 23:05:39.695096 11040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:39.728812 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37721
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:39.813974 11052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:39.814287 11052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:39.814316 11052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:39.815721 11052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:39.815784 11052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:39.815815 11052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:39.817628 11052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:39.817932 11052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:39.818190 11052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:39.823354 11060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:39.823454 11058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:39.823601 11052 server_base.cc:1034] running on GCE node
W20260923 23:05:39.823625 11057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:39.823985 11052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:39.824414 11052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:39.825621 11052 hybrid_clock.cc:648] HybridClock initialized: now 1790204739825605 us; error 39 us; skew 500 ppm
Sep 23 23:05:39 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204739, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:39.828469 11052 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:39.829077 11052 webserver.cc:460] Webserver started at http://127.5.74.1:34201/ using document root <none> and password file <none>
I20260923 23:05:39.829468 11052 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:39.829566 11052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:39.829728 11052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:39.830811 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "019667bb38c44d73a2c3a6e7da652d99"
format_stamp: "Formatted at 2026-09-23 23:05:39 on dist-test-slave-j4c5"
server_key: "418d76048efa22ddb81e72c9cc7516eb"
server_key_iv: "7ee64527dfc60ebbfbf35f872e97288d"
server_key_version: "encryptionkey@0"
I20260923 23:05:39.831171 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "019667bb38c44d73a2c3a6e7da652d99"
format_stamp: "Formatted at 2026-09-23 23:05:39 on dist-test-slave-j4c5"
server_key: "418d76048efa22ddb81e72c9cc7516eb"
server_key_iv: "7ee64527dfc60ebbfbf35f872e97288d"
server_key_version: "encryptionkey@0"
I20260923 23:05:39.832691 11052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:39.833456 11067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:39.833676 11052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:39.833784 11052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "019667bb38c44d73a2c3a6e7da652d99"
format_stamp: "Formatted at 2026-09-23 23:05:39 on dist-test-slave-j4c5"
server_key: "418d76048efa22ddb81e72c9cc7516eb"
server_key_iv: "7ee64527dfc60ebbfbf35f872e97288d"
server_key_version: "encryptionkey@0"
I20260923 23:05:39.833873 11052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:39.847352 11052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:39.849916 11052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:39.850097 11052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:39.850416 11052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:39.850734 11052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:39.850788 11052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:39.850831 11052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:39.850862 11052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:39.857331 11052 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:40889
I20260923 23:05:39.857412 11180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:40889 every 8 connection(s)
I20260923 23:05:39.857729 11052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:05:39 dist-test-slave-j4c5 krb5kdc[10945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204739, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:39.865079 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 11052
I20260923 23:05:39.865207 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:39.865413 5416 external_mini_cluster.cc:1428] Setting key 6ba75c2ea4d008f7923458e3e65f3cc1
I20260923 23:05:39.868582 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:39.858538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:52315 (local address 127.5.74.62:37721)
0923 23:05:39.858588 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:39.858590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:39.858778 (+ 188us) server_negotiation.cc:407] Connection header received
0923 23:05:39.859757 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:39.859758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:39.859770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:39.859849 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:39.860814 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:39.861375 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:39.862439 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:39.862651 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:39.862712 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:39.865319 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:39.865321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:39.865322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:39.865331 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:39.866838 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:39.867374 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:39.867375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:39.867375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:39.867441 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:39.867752 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:39.867753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:39.867754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:39.867941 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:39.868018 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:39.868366 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 23:05:39.868473 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:05:39.868736 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:39.858498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37721 (local address 127.5.74.1:52315)
0923 23:05:39.858647 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:39.858689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:39.859591 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:39.859946 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:39.859950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:39.860487 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:39.860697 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:39.860702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:39.861485 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:39.861485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:39.862322 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:39.862324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:39.862762 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:39.862763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:39.862891 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:39.863514 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:39.863523 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:39.865192 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:39.866931 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:39.866939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:39.866941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:39.867290 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:39.867512 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:39.867513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:39.867513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:39.867670 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:39.868020 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:39.868025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:39.868204 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 23:05:39.868324 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 23:05:39.868505 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260923 23:05:39.869356 11181 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37721
I20260923 23:05:39.869508 11181 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:39.869711 11181 heartbeater.cc:511] Master 127.5.74.62:37721 requested a full tablet report, sending...
I20260923 23:05:39.870328 10990 ts_manager.cc:194] Registered new tserver with Master: 019667bb38c44d73a2c3a6e7da652d99 (127.5.74.1:40889)
I20260923 23:05:39.871292 10990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:52315
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:39.897099 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37721
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:39.982338 11186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:39.982563 11186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:39.982590 11186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:39.983850 11186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:39.983906 11186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:39.983937 11186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:39.986274 11186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:39.986737 11186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:39.987092 11186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:39.993017 11194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:39.993072 11192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:39.993022 11191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:39.993494 11186 server_base.cc:1034] running on GCE node
I20260923 23:05:39.993778 11186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:39.994042 11186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:39.995260 11186 hybrid_clock.cc:648] HybridClock initialized: now 1790204739995240 us; error 29 us; skew 500 ppm
Sep 23 23:05:39 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204739, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:39.998134 11186 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:39.998554 11186 webserver.cc:460] Webserver started at http://127.5.74.2:44935/ using document root <none> and password file <none>
I20260923 23:05:39.998750 11186 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:39.998791 11186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:39.998884 11186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:40.000278 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "1d80f70fd26143108451a6def8f997b6"
format_stamp: "Formatted at 2026-09-23 23:05:39 on dist-test-slave-j4c5"
server_key: "f5f8f072f511ddaa645262908c224498"
server_key_iv: "c987a48e9ddffbf29692bede68af578f"
server_key_version: "encryptionkey@0"
I20260923 23:05:40.000743 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "1d80f70fd26143108451a6def8f997b6"
format_stamp: "Formatted at 2026-09-23 23:05:39 on dist-test-slave-j4c5"
server_key: "f5f8f072f511ddaa645262908c224498"
server_key_iv: "c987a48e9ddffbf29692bede68af578f"
server_key_version: "encryptionkey@0"
I20260923 23:05:40.002451 11186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:05:40.003484 11201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:40.003758 11186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:40.003863 11186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "1d80f70fd26143108451a6def8f997b6"
format_stamp: "Formatted at 2026-09-23 23:05:39 on dist-test-slave-j4c5"
server_key: "f5f8f072f511ddaa645262908c224498"
server_key_iv: "c987a48e9ddffbf29692bede68af578f"
server_key_version: "encryptionkey@0"
I20260923 23:05:40.003952 11186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:40.028120 11186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:40.030748 11186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:40.030912 11186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:40.031180 11186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:40.031504 11186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:40.031539 11186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:40.031563 11186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:40.031576 11186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:40.037964 11186 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:42837
I20260923 23:05:40.038025 11314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:42837 every 8 connection(s)
I20260923 23:05:40.038367 11186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:05:40.042984 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 11186
I20260923 23:05:40.043116 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:40.043294 5416 external_mini_cluster.cc:1428] Setting key dfd2da58df3bf7804e7848baa6086eb2
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204739, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:40.048846 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:40.039121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:42069 (local address 127.5.74.62:37721)
0923 23:05:40.039190 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:40.039192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:40.039435 (+ 243us) server_negotiation.cc:407] Connection header received
0923 23:05:40.040497 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:40.040498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:40.040511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:40.040560 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:40.041206 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.041744 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.042763 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.043055 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.043139 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:40.045801 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:40.045803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:40.045804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:40.045812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:40.047293 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.047764 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.047765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.047766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.047826 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.048101 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.048102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.048102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.048283 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:40.048359 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:40.048657 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 23:05:40.048746 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 23:05:40.049068 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:40.039094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37721 (local address 127.5.74.2:42069)
0923 23:05:40.039280 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:40.039317 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:40.040386 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:40.040644 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:40.040648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:40.040917 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:40.041112 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:40.041116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.041848 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.041849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:40.042658 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:40.042660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.043184 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.043185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:40.043380 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:40.044021 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:40.044032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:40.045697 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:40.047401 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:40.047409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:40.047412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:40.047669 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:40.047897 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:40.047897 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:40.047898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:40.048024 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:40.048350 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:40.048357 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:40.048516 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 23:05:40.048636 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 23:05:40.048810 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260923 23:05:40.049499 11315 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37721
I20260923 23:05:40.049595 11315 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:40.049798 11315 heartbeater.cc:511] Master 127.5.74.62:37721 requested a full tablet report, sending...
I20260923 23:05:40.050194 10990 ts_manager.cc:194] Registered new tserver with Master: 1d80f70fd26143108451a6def8f997b6 (127.5.74.2:42837)
I20260923 23:05:40.050592 10990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:42069
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:40.072706 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37721
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:40.166307 11320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:40.166548 11320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:40.166574 11320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:40.167837 11320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:40.167887 11320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:40.167914 11320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:40.169510 11320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:40.169768 11320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:40.170014 11320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:40.175372 11325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:40.175554 11328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:40.175580 11320 server_base.cc:1034] running on GCE node
W20260923 23:05:40.175372 11326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:40.175959 11320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:40.176286 11320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:40.177450 11320 hybrid_clock.cc:648] HybridClock initialized: now 1790204740177428 us; error 43 us; skew 500 ppm
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:40.180251 11320 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:40.180732 11320 webserver.cc:460] Webserver started at http://127.5.74.3:33175/ using document root <none> and password file <none>
I20260923 23:05:40.180964 11320 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:40.181030 11320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:40.181182 11320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:40.182236 11320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "fc0a766b4b064796b9783cc3234b0bd6"
format_stamp: "Formatted at 2026-09-23 23:05:40 on dist-test-slave-j4c5"
server_key: "33b094529cd9b5166e5e65e3f1910742"
server_key_iv: "15fd52888845fbc8daa3745315d28114"
server_key_version: "encryptionkey@0"
I20260923 23:05:40.182588 11320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "fc0a766b4b064796b9783cc3234b0bd6"
format_stamp: "Formatted at 2026-09-23 23:05:40 on dist-test-slave-j4c5"
server_key: "33b094529cd9b5166e5e65e3f1910742"
server_key_iv: "15fd52888845fbc8daa3745315d28114"
server_key_version: "encryptionkey@0"
I20260923 23:05:40.183940 11320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:05:40.184916 11335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:40.185117 11320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:40.185218 11320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "fc0a766b4b064796b9783cc3234b0bd6"
format_stamp: "Formatted at 2026-09-23 23:05:40 on dist-test-slave-j4c5"
server_key: "33b094529cd9b5166e5e65e3f1910742"
server_key_iv: "15fd52888845fbc8daa3745315d28114"
server_key_version: "encryptionkey@0"
I20260923 23:05:40.185294 11320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:40.210166 11320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:40.212674 11320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:40.212859 11320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:40.213117 11320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:40.213438 11320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:40.213490 11320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:40.213526 11320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:40.213567 11320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:40.219864 11320 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:41393
I20260923 23:05:40.219904 11448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:41393 every 8 connection(s)
I20260923 23:05:40.220342 11320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:40.229189 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 11320
I20260923 23:05:40.229295 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:40.229488 5416 external_mini_cluster.cc:1428] Setting key 199abe78b6f39f3c44744fc9dbbb2d68
I20260923 23:05:40.231541 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:40.221149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37721 (local address 127.5.74.3:48883)
0923 23:05:40.221370 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:40.221407 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:40.222529 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:40.222783 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:40.222786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:40.223064 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:40.223268 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:40.223272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.224015 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.224016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:40.224970 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:40.224973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.225412 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.225413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:40.225531 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:40.226221 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:40.226239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:40.227805 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:40.229835 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:40.229844 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:40.229846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:40.230180 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:40.230399 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:40.230400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:40.230400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:40.230537 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:40.230976 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:40.230983 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:40.231115 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 23:05:40.231221 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 23:05:40.231364 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":110,"threads_started":1}
I20260923 23:05:40.231554 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:40.221177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:48883 (local address 127.5.74.62:37721)
0923 23:05:40.221250 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:40.221253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:40.221492 (+ 239us) server_negotiation.cc:407] Connection header received
0923 23:05:40.222650 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:40.222651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:40.222662 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:40.222719 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:40.223374 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.223892 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.225072 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.225275 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.225338 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:40.227921 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:40.227923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:40.227924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:40.227941 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:40.229760 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.230250 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.230251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.230251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.230322 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.230618 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.230619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.230619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.230817 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:40.230977 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:40.231410 (+ 433us) server_negotiation.cc:299] Negotiation successful
0923 23:05:40.231491 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 23:05:40.232088 11449 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37721
I20260923 23:05:40.232234 11449 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:40.232429 11449 heartbeater.cc:511] Master 127.5.74.62:37721 requested a full tablet report, sending...
I20260923 23:05:40.232893 10990 ts_manager.cc:194] Registered new tserver with Master: fc0a766b4b064796b9783cc3234b0bd6 (127.5.74.3:41393)
I20260923 23:05:40.233305 10990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:48883
I20260923 23:05:40.242000 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/test-admin.keytab.
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:40.261265 5416 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 23:05:40.263236 11456 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:40.262220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.5.74.1:40889)
0923 23:05:40.262419 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:40.262425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:40.262473 (+ 48us) server_negotiation.cc:407] Connection header received
0923 23:05:40.262509 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:40.262511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:40.262526 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:40.262601 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:40.263147 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47940: BlockingRecv error: recv got EOF from 127.0.0.1:47940 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
W20260923 23:05:40.263334 5416 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5cc' of type FILE
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:40.264484 5416 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:40.265411 5416 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:40.275363 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:40.266013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.5.74.1:40889)
0923 23:05:40.266078 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:40.266080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:40.266111 (+ 31us) server_negotiation.cc:407] Connection header received
0923 23:05:40.266206 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:40.266207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:40.266221 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:40.266276 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:40.266754 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.267576 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.268564 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.268925 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.269002 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:40.271944 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:40.271957 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:40.271957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:40.271966 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:40.273878 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.274269 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.274273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.274273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.274381 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.274622 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.274623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.274624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.274786 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:40.274910 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:40.275088 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 23:05:40.275212 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260923 23:05:40.277644 11456 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:40.277042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.5.74.1:40889)
0923 23:05:40.277078 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:40.277080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:40.277111 (+ 31us) server_negotiation.cc:407] Connection header received
0923 23:05:40.277161 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:40.277162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:40.277171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:40.277215 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:40.277588 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47960: BlockingRecv error: recv got EOF from 127.0.0.1:47960 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 23:05:40.277705 5416 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5cc' of type FILE
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:40.278719 5416 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:40.279608 5416 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:40.287607 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:40.280300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.5.74.1:40889)
0923 23:05:40.280401 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:40.280402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:40.280414 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:05:40.280433 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:40.280434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:40.280442 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:40.280501 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:40.280972 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.281487 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.282212 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.282432 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.282494 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:40.284943 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:40.284945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:40.284946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:40.284954 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:40.286436 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.286807 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.286809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.286809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.286877 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.287082 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.287083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.287083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.287189 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:40.287269 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:40.287424 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 23:05:40.287492 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 23:05:40.289727 11456 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:40.288964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.5.74.1:40889)
0923 23:05:40.289051 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:40.289053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:40.289169 (+ 116us) server_negotiation.cc:407] Connection header received
0923 23:05:40.289262 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:40.289262 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:40.289272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:40.289345 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:40.289677 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47980: BlockingRecv error: recv got EOF from 127.0.0.1:47980 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 23:05:40.289928 5416 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204714633713-5416-0/krb5kdc/krb5cc' of type FILE
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:40.291060 5416 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:40.292128 5416 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[10945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:05:40.299844 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:40.292724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.5.74.1:40889)
0923 23:05:40.292789 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:40.292791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:40.292899 (+ 108us) server_negotiation.cc:407] Connection header received
0923 23:05:40.292977 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:40.292978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:40.292986 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:40.293057 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:40.293444 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.293917 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.294621 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:40.294819 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:40.294938 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:40.297311 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:40.297313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:40.297313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:40.297321 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:40.298676 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.299062 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.299063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.299064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.299125 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:40.299328 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:40.299329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:40.299330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:40.299437 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:40.299540 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:40.299654 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 23:05:40.299732 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 23:05:40.300791 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 11052
I20260923 23:05:40.305658 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 11186
I20260923 23:05:40.310586 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 11320
I20260923 23:05:40.317965 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 10958
2026-09-23T23:05:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2024 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[11469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[11469](info): set up 2 sockets
Sep 23 23:05:40 dist-test-slave-j4c5 krb5kdc[11469](info): commencing operation
krb5kdc: starting...
W20260923 23:05:41.409000 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:41 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.62.keytab.
Entry for principal oryx/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.62.keytab.
I20260923 23:05:41.552542 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:38663
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.62.keytab
--principal=oryx/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:38663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:41.648276 11482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:41.648581 11482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:41.648625 11482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:41.649863 11482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:41.649940 11482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:41.649962 11482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:41.650111 11482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:41.650147 11482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:41.651842 11482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:38663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.62.keytab
--principal=oryx/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:38663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:41.652185 11482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:41.652449 11482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:41.658387 11490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:41.658404 11488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:41.658416 11487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:41.658964 11482 server_base.cc:1034] running on GCE node
I20260923 23:05:41.659279 11482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:41.659615 11482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:41.660813 11482 hybrid_clock.cc:648] HybridClock initialized: now 1790204741660858 us; error 108 us; skew 500 ppm
Sep 23 23:05:41 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204741, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:41.663772 11482 init.cc:377] Logged in from keytab as oryx/127.5.74.62@KRBTEST.COM (short username oryx)
I20260923 23:05:41.664363 11482 webserver.cc:460] Webserver started at http://127.5.74.62:38577/ using document root <none> and password file <none>
I20260923 23:05:41.664602 11482 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:41.664671 11482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:41.664824 11482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:41.666283 11482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "fb622f88b87c4f2c8c62912ca7a019e7"
format_stamp: "Formatted at 2026-09-23 23:05:41 on dist-test-slave-j4c5"
server_key: "06aa19a8bd11948f873f1a8857f007e3"
server_key_iv: "215807cb60e3443774df9ec12e504c58"
server_key_version: "encryptionkey@0"
I20260923 23:05:41.666733 11482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "fb622f88b87c4f2c8c62912ca7a019e7"
format_stamp: "Formatted at 2026-09-23 23:05:41 on dist-test-slave-j4c5"
server_key: "06aa19a8bd11948f873f1a8857f007e3"
server_key_iv: "215807cb60e3443774df9ec12e504c58"
server_key_version: "encryptionkey@0"
I20260923 23:05:41.668396 11482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260923 23:05:41.669356 11497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:41.669565 11482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:41.669677 11482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "fb622f88b87c4f2c8c62912ca7a019e7"
format_stamp: "Formatted at 2026-09-23 23:05:41 on dist-test-slave-j4c5"
server_key: "06aa19a8bd11948f873f1a8857f007e3"
server_key_iv: "215807cb60e3443774df9ec12e504c58"
server_key_version: "encryptionkey@0"
I20260923 23:05:41.669768 11482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:41.700063 11482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:41.702872 11482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:41.703071 11482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:41.707595 11482 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:38663
I20260923 23:05:41.707626 11549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:38663 every 8 connection(s)
I20260923 23:05:41.708030 11482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:41.709185 11550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:41.709357 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 11482
I20260923 23:05:41.709510 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:41.709743 5416 external_mini_cluster.cc:1428] Setting key 2c803382973bbea5ad1530a27dda2dc9
I20260923 23:05:41.712980 11550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: Bootstrap starting.
I20260923 23:05:41.713713 11550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:41.714026 11550 log.cc:824] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:41.714874 11550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: No bootstrap required, opened a new log
I20260923 23:05:41.717331 11550 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } }
I20260923 23:05:41.717577 11550 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:41.717617 11550 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb622f88b87c4f2c8c62912ca7a019e7, State: Initialized, Role: FOLLOWER
Sep 23 23:05:41 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:41.717752 11550 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } }
I20260923 23:05:41.717887 11550 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:41.718003 11550 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:41.718091 11550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:41.718890 11550 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } }
I20260923 23:05:41.719094 11550 leader_election.cc:302] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb622f88b87c4f2c8c62912ca7a019e7; no voters:
I20260923 23:05:41.719302 11550 leader_election.cc:288] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:41.719384 11555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:41.719646 11555 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 1 LEADER]: Becoming Leader. State: Replica: fb622f88b87c4f2c8c62912ca7a019e7, State: Running, Role: LEADER
I20260923 23:05:41.719764 11550 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:41.719794 11555 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } }
I20260923 23:05:41.720243 11555 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb622f88b87c4f2c8c62912ca7a019e7. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } }
I20260923 23:05:41.720371 11555 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:41.720647 11562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:41.721552 11562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:41.722097 11557 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } }
I20260923 23:05:41.722307 11557 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:41.723555 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:41.710392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.5.74.62:38663)
0923 23:05:41.710578 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:41.710585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:41.710613 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:05:41.711067 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:41.711074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:41.711124 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:41.711266 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:41.712266 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:41.713362 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:41.714424 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:41.714910 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:41.715046 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:41.718317 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:41.718322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:41.718322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:41.718335 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:41.721184 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:41.721809 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:41.721813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:41.721813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:41.721995 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:41.722262 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:41.722263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:41.722263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:41.722585 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:41.722758 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:41.723018 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 23:05:41.723172 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260923 23:05:41.723580 11562 catalog_manager.cc:1347] Generated new cluster ID: 392d22574c744aaa838e1b7ef1304bff
I20260923 23:05:41.723629 11562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:41.737210 11562 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:41.738076 11562 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:41.741519 11562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: Generated new TSK 0
I20260923 23:05:41.741712 11562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab.
Entry for principal oryx/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab.
I20260923 23:05:41.776810 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:38663
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab
--principal=oryx/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:41.866103 11576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:41.866361 11576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:41.866405 11576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:41.867771 11576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:41.868049 11576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:41.868111 11576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:41.870283 11576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab
--principal=oryx/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:41.870602 11576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:41.870855 11576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:41.877274 11584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:41.877296 11581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:41.877497 11582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:41.877722 11576 server_base.cc:1034] running on GCE node
I20260923 23:05:41.878036 11576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:41.878497 11576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:41.879705 11576 hybrid_clock.cc:648] HybridClock initialized: now 1790204741879661 us; error 68 us; skew 500 ppm
Sep 23 23:05:41 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204741, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:41.882669 11576 init.cc:377] Logged in from keytab as oryx/127.5.74.1@KRBTEST.COM (short username oryx)
I20260923 23:05:41.883189 11576 webserver.cc:460] Webserver started at http://127.5.74.1:35005/ using document root <none> and password file <none>
I20260923 23:05:41.883405 11576 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:41.883452 11576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:41.883545 11576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:41.884650 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "61144ca035074cf6b49ac17c874d9d42"
format_stamp: "Formatted at 2026-09-23 23:05:41 on dist-test-slave-j4c5"
server_key: "ab1d6ec7639fd1e6eb52d6e3510689c6"
server_key_iv: "84c3d5dcadb760600dcd22173c51a991"
server_key_version: "encryptionkey@0"
I20260923 23:05:41.884958 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "61144ca035074cf6b49ac17c874d9d42"
format_stamp: "Formatted at 2026-09-23 23:05:41 on dist-test-slave-j4c5"
server_key: "ab1d6ec7639fd1e6eb52d6e3510689c6"
server_key_iv: "84c3d5dcadb760600dcd22173c51a991"
server_key_version: "encryptionkey@0"
I20260923 23:05:41.886299 11576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:41.887226 11591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:41.887501 11576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:41.887585 11576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "61144ca035074cf6b49ac17c874d9d42"
format_stamp: "Formatted at 2026-09-23 23:05:41 on dist-test-slave-j4c5"
server_key: "ab1d6ec7639fd1e6eb52d6e3510689c6"
server_key_iv: "84c3d5dcadb760600dcd22173c51a991"
server_key_version: "encryptionkey@0"
I20260923 23:05:41.887643 11576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:41.902092 11576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:41.904673 11576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:41.904808 11576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:41.905023 11576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:41.905297 11576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:41.905328 11576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:41.905351 11576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:41.905364 11576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:41.911523 11576 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:46765
I20260923 23:05:41.911584 11704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:46765 every 8 connection(s)
I20260923 23:05:41.911958 11576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:41.913307 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 11576
I20260923 23:05:41.913450 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:41.913648 5416 external_mini_cluster.cc:1428] Setting key 813744ed49b5fbccc178fcc97b2ca3ec
Sep 23 23:05:41 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204741, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:05:41.925160 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:41.912933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:53355 (local address 127.5.74.62:38663)
0923 23:05:41.913056 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:41.913059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:41.913265 (+ 206us) server_negotiation.cc:407] Connection header received
0923 23:05:41.913823 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:41.913825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:41.913838 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:41.913920 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:41.915344 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:41.916352 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:41.918126 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:41.918334 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:41.918417 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:41.921391 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:41.921393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:41.921394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:41.921403 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:41.923201 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:41.923792 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:41.923794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:41.923794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:41.923859 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:41.924311 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:41.924312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:41.924313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:41.924474 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:41.924559 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:41.924874 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 23:05:41.925054 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 23:05:41.925390 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:41.912850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.1:53355)
0923 23:05:41.913083 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:41.913128 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:41.913707 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:41.914004 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:41.914009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:41.914800 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:41.915005 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:41.915011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:41.916478 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:41.916480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:41.918012 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:41.918016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:41.918437 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:41.918439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:41.918600 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:41.919472 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:41.919484 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:41.921282 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:41.923373 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:41.923382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:41.923385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:41.923711 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:41.923934 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:41.923935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:41.923935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:41.924239 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:41.924549 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:41.924556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:41.924715 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 23:05:41.924838 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 23:05:41.925126 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":143,"threads_started":1}
I20260923 23:05:41.926231 11705 heartbeater.cc:347] Connected to a master server at 127.5.74.62:38663
I20260923 23:05:41.926422 11705 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:41.926659 11705 heartbeater.cc:511] Master 127.5.74.62:38663 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.2@KRBTEST.COM" created.
I20260923 23:05:41.927675 11514 ts_manager.cc:194] Registered new tserver with Master: 61144ca035074cf6b49ac17c874d9d42 (127.5.74.1:46765)
I20260923 23:05:41.929049 11514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:53355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab.
Entry for principal oryx/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab.
I20260923 23:05:41.949239 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:38663
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab
--principal=oryx/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:42.035441 11710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:42.035694 11710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:42.035732 11710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:42.037045 11710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:42.037117 11710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:42.037165 11710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:42.038762 11710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab
--principal=oryx/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:42.039054 11710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:42.039306 11710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:42.044668 11716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.044668 11718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.044675 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:42.044700 11710 server_base.cc:1034] running on GCE node
I20260923 23:05:42.045153 11710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:42.045415 11710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:42.046590 11710 hybrid_clock.cc:648] HybridClock initialized: now 1790204742046578 us; error 30 us; skew 500 ppm
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:42.049441 11710 init.cc:377] Logged in from keytab as oryx/127.5.74.2@KRBTEST.COM (short username oryx)
I20260923 23:05:42.049957 11710 webserver.cc:460] Webserver started at http://127.5.74.2:41311/ using document root <none> and password file <none>
I20260923 23:05:42.050189 11710 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:42.050256 11710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:42.050419 11710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:42.051537 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "1bc9bd0f474f425895102fae742d8082"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "5cb3028f53455b2cfa4aac0c86d91d9c"
server_key_iv: "42255bc999f16f92dc4cf0f81a1d1377"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.052050 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "1bc9bd0f474f425895102fae742d8082"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "5cb3028f53455b2cfa4aac0c86d91d9c"
server_key_iv: "42255bc999f16f92dc4cf0f81a1d1377"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.053614 11710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:42.054610 11725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.054886 11710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:42.055018 11710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "1bc9bd0f474f425895102fae742d8082"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "5cb3028f53455b2cfa4aac0c86d91d9c"
server_key_iv: "42255bc999f16f92dc4cf0f81a1d1377"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.055109 11710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:42.089946 11710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:42.093395 11710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:42.093603 11710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:42.093892 11710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:42.094262 11710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:42.094323 11710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.094376 11710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:42.094411 11710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.102923 11710 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:43089
I20260923 23:05:42.102974 11838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:43089 every 8 connection(s)
I20260923 23:05:42.103353 11710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:05:42.105393 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 11710
I20260923 23:05:42.105484 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:42.105723 5416 external_mini_cluster.cc:1428] Setting key 769928a5796f7106d0608626acf337b6
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.5.74.3@KRBTEST.COM; defaulting to no policy
I20260923 23:05:42.115415 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.104322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:49075 (local address 127.5.74.62:38663)
0923 23:05:42.104391 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.104393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.104520 (+ 127us) server_negotiation.cc:407] Connection header received
0923 23:05:42.105086 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.105087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.105100 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.105154 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.106411 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.107164 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.108371 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.108576 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.108666 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.111229 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.111232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.111232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.111241 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.113084 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.113855 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.113857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.113857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.113924 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.114510 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.114513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.114513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.114662 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.114758 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.115209 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.115318 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:05:42.115626 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.104144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:49075)
0923 23:05:42.104374 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.104425 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.104976 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.105238 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.105242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.106005 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.106287 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.106293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.107286 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.107287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.108264 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.108267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.108679 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.108679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.108815 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.109499 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.109512 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.111110 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.113230 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.113240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.113243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.113718 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.114010 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.114045 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.114047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.114193 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.114747 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.114754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.115048 (+ 294us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.115184 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.115375 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260923 23:05:42.116523 11839 heartbeater.cc:347] Connected to a master server at 127.5.74.62:38663
I20260923 23:05:42.116631 11839 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:42.116863 11839 heartbeater.cc:511] Master 127.5.74.62:38663 requested a full tablet report, sending...
I20260923 23:05:42.117443 11514 ts_manager.cc:194] Registered new tserver with Master: 1bc9bd0f474f425895102fae742d8082 (127.5.74.2:43089)
I20260923 23:05:42.117903 11514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:49075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab.
Entry for principal oryx/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab.
I20260923 23:05:42.137176 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:38663
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab
--principal=oryx/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:42.228935 11844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:42.229176 11844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:42.229203 11844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:42.230377 11844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:42.230419 11844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:42.230449 11844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:42.231920 11844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab
--principal=oryx/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:42.232263 11844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:42.232483 11844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:42.238476 11850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.238329 11849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.238329 11852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:42.239076 11844 server_base.cc:1034] running on GCE node
I20260923 23:05:42.239291 11844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:42.239529 11844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:42.240715 11844 hybrid_clock.cc:648] HybridClock initialized: now 1790204742240697 us; error 30 us; skew 500 ppm
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:42.244112 11844 init.cc:377] Logged in from keytab as oryx/127.5.74.3@KRBTEST.COM (short username oryx)
I20260923 23:05:42.244670 11844 webserver.cc:460] Webserver started at http://127.5.74.3:41585/ using document root <none> and password file <none>
I20260923 23:05:42.244918 11844 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:42.244966 11844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:42.245059 11844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:42.246168 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "cd81e0e2f06a45d0beaae3377efa1dc2"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "dc29ef22996ef59db663020b7ec86cea"
server_key_iv: "34ef84ed2f08934485ce0763051bdaa4"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.246497 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "cd81e0e2f06a45d0beaae3377efa1dc2"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "dc29ef22996ef59db663020b7ec86cea"
server_key_iv: "34ef84ed2f08934485ce0763051bdaa4"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.247874 11844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:05:42.249009 11859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.249224 11844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:42.249310 11844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "cd81e0e2f06a45d0beaae3377efa1dc2"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "dc29ef22996ef59db663020b7ec86cea"
server_key_iv: "34ef84ed2f08934485ce0763051bdaa4"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.249364 11844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:42.262903 11844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:42.265548 11844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:42.265673 11844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:42.265884 11844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:42.266150 11844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:42.266179 11844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.266202 11844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:42.266214 11844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.272754 11844 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:35743
I20260923 23:05:42.272786 11972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:35743 every 8 connection(s)
I20260923 23:05:42.273281 11844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:42.283228 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 11844
I20260923 23:05:42.283476 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:42.283739 5416 external_mini_cluster.cc:1428] Setting key f603c508b344dfb79c49282154e246c0
I20260923 23:05:42.283968 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.274059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:38191 (local address 127.5.74.62:38663)
0923 23:05:42.274104 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.274106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.274286 (+ 180us) server_negotiation.cc:407] Connection header received
0923 23:05:42.274909 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.274911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.274923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.274983 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.275962 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.276531 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.277797 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.278014 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.278139 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.280822 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.280824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.280824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.280833 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.282337 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.282851 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.282853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.282853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.282920 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.283199 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.283200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.283200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.283361 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.283434 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.283780 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.283866 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:05:42.284274 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.274000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.3:38191)
0923 23:05:42.274164 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.274207 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.274781 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.275069 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.275073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.275645 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.275852 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.275856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.276645 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.276646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.277471 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.277473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.278152 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.278154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.278307 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.278977 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.278987 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.280677 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.282481 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.282491 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.282494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.282764 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.282990 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.282991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.282991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.283120 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.283425 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.283431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.283617 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.283759 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.283951 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260923 23:05:42.284654 11973 heartbeater.cc:347] Connected to a master server at 127.5.74.62:38663
I20260923 23:05:42.284775 11973 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:42.284996 11973 heartbeater.cc:511] Master 127.5.74.62:38663 requested a full tablet report, sending...
I20260923 23:05:42.285459 11514 ts_manager.cc:194] Registered new tserver with Master: cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743)
I20260923 23:05:42.286031 11514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:38191
I20260923 23:05:42.296429 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.74.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.61.keytab.
Entry for principal oryx/127.5.74.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.61.keytab.
I20260923 23:05:42.327584 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.61:38741
--webserver_interface=127.5.74.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.61.keytab
--principal=oryx/127.5.74.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:38663,127.5.74.61:38741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:42.417307 11978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:42.417551 11978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:42.417578 11978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:42.418906 11978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:42.418963 11978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:42.418982 11978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:42.418996 11978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:42.419010 11978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:42.421244 11978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:38663,127.5.74.61:38741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.61.keytab
--principal=oryx/127.5.74.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.61:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.74.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
W20260923 23:05:42.421370 11978 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.74.62:38663,127.5.74.61:38741'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 23:05:42.421609 11978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:42.421878 11978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:42.427716 11984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.427697 11986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.427697 11983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:42.428048 11978 server_base.cc:1034] running on GCE node
I20260923 23:05:42.428335 11978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:42.428663 11978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:42.429826 11978 hybrid_clock.cc:648] HybridClock initialized: now 1790204742429802 us; error 32 us; skew 500 ppm
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:42.434231 11978 init.cc:377] Logged in from keytab as oryx/127.5.74.61@KRBTEST.COM (short username oryx)
I20260923 23:05:42.434826 11978 webserver.cc:460] Webserver started at http://127.5.74.61:39741/ using document root <none> and password file <none>
I20260923 23:05:42.435106 11978 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:42.435158 11978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:42.435302 11978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:42.436508 11978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data/instance:
uuid: "3efbc7de087342ba8b39f238f8c23a37"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "2ffda0b1b47106caf748b03170a3e757"
server_key_iv: "d755a2e24a89847085b7a80e2a487d94"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.436964 11978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal/instance:
uuid: "3efbc7de087342ba8b39f238f8c23a37"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "2ffda0b1b47106caf748b03170a3e757"
server_key_iv: "d755a2e24a89847085b7a80e2a487d94"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.438511 11978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:42.439812 11993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.440219 11978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:42.440327 11978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal
uuid: "3efbc7de087342ba8b39f238f8c23a37"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "2ffda0b1b47106caf748b03170a3e757"
server_key_iv: "d755a2e24a89847085b7a80e2a487d94"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.440397 11978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:42.454638 11978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:42.457603 11978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:42.457731 11978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:42.462651 11978 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.61:38741
I20260923 23:05:42.462718 12045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.61:38741 every 8 connection(s)
I20260923 23:05:42.463052 11978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
I20260923 23:05:42.463573 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 11978
I20260923 23:05:42.464434 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:42.466568 12046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.74.61@KRBTEST.COM
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.61@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:42.476469 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.467326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.5.74.62:38663)
0923 23:05:42.467393 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.467395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.467709 (+ 314us) server_negotiation.cc:407] Connection header received
0923 23:05:42.467740 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.467741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.467751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.467802 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.468493 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.469307 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.470489 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.470674 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.470750 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.473479 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.473482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.473482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.473491 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.474992 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.475430 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.475431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.475432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.475481 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.475718 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.475719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.475719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.475871 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.475933 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.476276 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.476361 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:05:42.476603 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.467237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:59696)
0923 23:05:42.467550 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.467600 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.467687 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.467834 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.467836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.468200 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.468282 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.468285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.469422 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.469423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.470367 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.470371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.470771 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.470772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.470860 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.471483 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.471491 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.473311 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.475091 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.475097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.475100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.475312 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.475548 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.475549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.475549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.475645 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.475947 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.475952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.476025 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.476129 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.476323 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":99,"threads_started":1}
I20260923 23:05:42.476763 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.464294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.5.74.61:38741)
0923 23:05:42.464473 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.464481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.464533 (+ 52us) server_negotiation.cc:407] Connection header received
0923 23:05:42.464996 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.465002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.465049 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.465167 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.465997 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.467165 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.468490 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.469267 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.469414 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.472961 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.472969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.472970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.472978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.474988 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.475393 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.475397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.475397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.475498 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.475727 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.475728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.475729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.476018 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.476196 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.476505 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.476595 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":97,"threads_started":1}
I20260923 23:05:42.478315 12046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.61@KRBTEST.COM for oryx/127.5.74.61@KRBTEST.COM
I20260923 23:05:42.486281 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.478657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.0.0.1:58872)
0923 23:05:42.478737 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.478749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.478799 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.478911 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.478912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.479036 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.479094 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.479095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.479823 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.479824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.480425 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.480427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.480973 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.480974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.481041 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.481560 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.481564 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.482964 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.485210 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.485211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.485212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.485396 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.485677 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.485677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.485678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.485735 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.486030 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.486030 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.486067 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.486132 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.486209 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 23:05:42.486441 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.478738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.5.74.61:38741)
0923 23:05:42.478799 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.478801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.478810 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:42.478854 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.478855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.478863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.478908 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.479175 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.479686 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.480509 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.480848 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.480956 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.483052 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.483054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.483054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.483062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.485087 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.485529 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.485530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.485530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.485583 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.485830 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.485831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.485832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.485948 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.486011 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.486185 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.486262 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 23:05:42.488263 12046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: Bootstrap starting.
I20260923 23:05:42.489424 12046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:42.489732 12046 log.cc:824] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:42.490963 12046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: No bootstrap required, opened a new log
I20260923 23:05:42.492956 12046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } }
I20260923 23:05:42.493132 12046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:42.493180 12046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efbc7de087342ba8b39f238f8c23a37, State: Initialized, Role: FOLLOWER
I20260923 23:05:42.493360 12046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } }
I20260923 23:05:42.493661 12052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } } }
I20260923 23:05:42.493726 12046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:42.493750 12052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:05:42.495653 12063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:05:42.495916 12063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:05:42.498631 11978 master_runner.cc:428] Detected that this master 3efbc7de087342ba8b39f238f8c23a37 is joining an existing cluster
I20260923 23:05:42.498682 11978 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.74.61:38741
I20260923 23:05:42.508872 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.499817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.0.0.1:58882)
0923 23:05:42.500247 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.500256 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.500332 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.500471 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.500472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.500694 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.500752 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.500754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.501687 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.501688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.503094 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.503097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.503858 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.503860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.503956 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.504779 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.504785 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.505558 (+ 773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.507558 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.507559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.507559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.507733 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.507947 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.507948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.507948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.508013 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.508360 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.508361 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.508399 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.508487 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.508737 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":104,"threads_started":1}
I20260923 23:05:42.509294 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.500006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.5.74.61:38741)
0923 23:05:42.500069 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.500071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.500319 (+ 248us) server_negotiation.cc:407] Connection header received
0923 23:05:42.500394 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.500395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.500405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.500466 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.500835 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.501592 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.503191 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.503728 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.503857 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.505676 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.505678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.505678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.505687 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.507464 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.507818 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.507820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.507820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.507875 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.508083 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.508084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.508084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.508251 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.508338 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.508515 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.509210 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":523,"server-negotiator.queue_time_us":43}
I20260923 23:05:42.509853 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.499881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.5.74.62:38663)
0923 23:05:42.499936 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.499938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.500018 (+ 80us) server_negotiation.cc:407] Connection header received
0923 23:05:42.500139 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.500141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.500176 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.500247 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.500971 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.501716 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.502522 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.502737 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.502830 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.504523 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.504526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.504527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.504541 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.507084 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.508325 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.508327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.508327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.508392 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.508721 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.508723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.508723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.508885 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.508973 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.509664 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.509765 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:05:42.510190 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.499763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:59702)
0923 23:05:42.499943 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.499956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.500053 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.500318 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.500320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.500583 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.500879 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.500881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.501791 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.501792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.502409 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.502412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.502844 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.502845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.502910 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.503833 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.503839 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.504360 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.507204 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.507205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.507206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.508228 (+ 1022us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.508469 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.508470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.508471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.508543 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.509303 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.509304 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.509362 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.509939 (+ 577us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.510070 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204741, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.62@KRBTEST.COM for oryx/127.5.74.61@KRBTEST.COM
I20260923 23:05:42.522919 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.513336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.5.74.61:38741)
0923 23:05:42.513400 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.513403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.513603 (+ 200us) server_negotiation.cc:407] Connection header received
0923 23:05:42.513661 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.513662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.513671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.513728 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.514366 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.514869 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.516135 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.516432 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.516507 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.519039 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.519041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.519041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.519050 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.521522 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.522011 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.522013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.522013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.522075 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.522290 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.522291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.522291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.522419 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.522491 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.522737 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.522817 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 23:05:42.523036 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.513285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.0.0.1:58884)
0923 23:05:42.513455 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.513501 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.513583 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.513735 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.513738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.514123 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.514219 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.514223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.514953 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.514954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.516031 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.516034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.516509 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.516510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.516610 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.517161 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.517166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.518925 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.521628 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.521641 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.521644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.521911 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.522145 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.522146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.522146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.522213 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.522495 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.522556 (+ 61us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.522615 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.522712 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.522910 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":113,"threads_started":1}
I20260923 23:05:42.523690 11514 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.74.61:38741: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37"
member_type: NON_VOTER
last_known_addr {
host: "127.5.74.61"
port: 38741
}
attrs {
promote: true
}
}
dest_uuid: "fb622f88b87c4f2c8c62912ca7a019e7"
cas_config_opid_index: -1
I20260923 23:05:42.524056 11514 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } }
I20260923 23:05:42.524878 12073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } }
I20260923 23:05:42.525040 12017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:42.531633 12017 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Refusing update from remote peer fb622f88b87c4f2c8c62912ca7a019e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 23:05:42.531682 12073 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.74.61:38741
I20260923 23:05:42.532395 12072 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } }
I20260923 23:05:42.532668 12072 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:42.533351 11978 master.cc:537] Master@127.5.74.61:38741 shutting down...
I20260923 23:05:42.534215 12072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 23:05:42.535447 11499 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 23:05:42.536715 11499 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 -> Peer 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61:38741): Couldn't send request to peer 3efbc7de087342ba8b39f238f8c23a37. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 23:05:42.539573 11978 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:42.539692 11978 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:42.539788 11978 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: stopping tablet replica
I20260923 23:05:42.543094 11978 master.cc:552] Master@127.5.74.61:38741 shutdown complete.
I20260923 23:05:42.543205 11978 master_runner.cc:305] Clearing existing system tablet
I20260923 23:05:42.543370 11978 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:42.543443 11978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:42.544260 11978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
WARNING: no policy specified for oryx/127.5.74.60@KRBTEST.COM; defaulting to no policy
I20260923 23:05:42.546677 12078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.546850 11978 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:42.546906 11978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260923 23:05:42.546924 11978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal
uuid: "3efbc7de087342ba8b39f238f8c23a37"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "2ffda0b1b47106caf748b03170a3e757"
server_key_iv: "d755a2e24a89847085b7a80e2a487d94"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.547640 11978 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:42.569384 11978 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 23:05:42.569535 11978 log.cc:1196] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 23:05:42.569815 11978 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: Deleting consensus metadata
I20260923 23:05:42.569988 11978 master_runner.cc:315] Copying system tablet from 127.5.74.62:38663
I20260923 23:05:42.570797 11978 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: tablet copy: Beginning tablet copy session from remote peer at address 127.5.74.62:38663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.60@KRBTEST.COM" created.
I20260923 23:05:42.579619 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.571303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.5.74.62:38663)
0923 23:05:42.571357 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.571359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.571877 (+ 518us) server_negotiation.cc:407] Connection header received
0923 23:05:42.571915 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.571916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.571928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.572068 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.572754 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.573548 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.574868 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.575182 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.575260 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.576571 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.576573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.576573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.576581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.578128 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.578657 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.578659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.578659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.578731 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.578964 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.578965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.578965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.579153 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.579240 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.579418 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.579509 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 23:05:42.579623 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.571212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:59708)
0923 23:05:42.571346 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.571365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.571428 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.572112 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.572113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.572561 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.572645 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.572648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.573693 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.573694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.574760 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.574763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.575319 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.575320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.575398 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.576007 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.576013 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.576454 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.578321 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.578322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.578323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.578577 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.578809 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.578810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.578810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.578891 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.579237 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.579238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.579295 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.579392 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.579511 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
I20260923 23:05:42.580438 11534 tablet_copy_service.cc:140] P fb622f88b87c4f2c8c62912ca7a019e7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3efbc7de087342ba8b39f238f8c23a37 ({username='oryx', principal='oryx/127.5.74.61@KRBTEST.COM'} at 127.0.0.1:59708)
I20260923 23:05:42.580554 11534 tablet_copy_service.cc:161] P fb622f88b87c4f2c8c62912ca7a019e7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3efbc7de087342ba8b39f238f8c23a37 at {username='oryx', principal='oryx/127.5.74.61@KRBTEST.COM'} at 127.0.0.1:59708: session id = 3efbc7de087342ba8b39f238f8c23a37-00000000000000000000000000000000
I20260923 23:05:42.581212 11534 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 23:05:42.581856 11978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 23:05:42.582978 11978 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: tablet copy: Starting download of 0 data blocks...
I20260923 23:05:42.583175 11978 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: tablet copy: Starting download of 1 WAL segments...
I20260923 23:05:42.584126 11978 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 23:05:42.586066 11534 tablet_copy_service.cc:342] P fb622f88b87c4f2c8c62912ca7a019e7: Request end of tablet copy session 3efbc7de087342ba8b39f238f8c23a37-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.74.61@KRBTEST.COM'} at 127.0.0.1:59708
I20260923 23:05:42.586148 11534 tablet_copy_service.cc:433] P fb622f88b87c4f2c8c62912ca7a019e7: ending tablet copy session 3efbc7de087342ba8b39f238f8c23a37-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3efbc7de087342ba8b39f238f8c23a37
I20260923 23:05:42.589399 11978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:42.590777 12086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.590777 12089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.590862 12087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:42.591385 11978 server_base.cc:1034] running on GCE node
I20260923 23:05:42.591504 11978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:42.591677 11978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:42.592821 11978 hybrid_clock.cc:648] HybridClock initialized: now 1790204742592839 us; error 56 us; skew 500 ppm
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:42.594385 11978 init.cc:377] Logged in from keytab as oryx/127.5.74.61@KRBTEST.COM (short username oryx)
I20260923 23:05:42.594854 11978 webserver.cc:460] Webserver started at http://127.5.74.61:45673/ using document root <none> and password file <none>
I20260923 23:05:42.595014 11978 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:42.595060 11978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.60.keytab.
Entry for principal oryx/127.5.74.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.60.keytab.
I20260923 23:05:42.595744 11978 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:42.596345 12096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.596475 11978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.596521 11978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal
uuid: "3efbc7de087342ba8b39f238f8c23a37"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "2ffda0b1b47106caf748b03170a3e757"
server_key_iv: "d755a2e24a89847085b7a80e2a487d94"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.596585 11978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:42.596707 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.60:40011
--webserver_interface=127.5.74.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.60.keytab
--principal=oryx/127.5.74.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:38663,127.5.74.61:38741,127.5.74.60:40011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:05:42.606801 11978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:42.608880 11978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:42.612236 11978 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.61:38741
I20260923 23:05:42.612327 12149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.61:38741 every 8 connection(s)
I20260923 23:05:42.612775 11978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
I20260923 23:05:42.613668 12150 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:05:42.614166 12150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: Bootstrap starting.
I20260923 23:05:42.617359 12150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:05:42.617710 12150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: Bootstrap complete.
I20260923 23:05:42.617962 12150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } }
I20260923 23:05:42.618067 12150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efbc7de087342ba8b39f238f8c23a37, State: Initialized, Role: LEARNER
I20260923 23:05:42.618108 12150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } }
I20260923 23:05:42.618311 12150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:42.618332 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } }
I20260923 23:05:42.618536 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: This master's current role is: LEARNER
I20260923 23:05:42.620541 12164 catalog_manager.cc:1259] Loaded cluster ID: 392d22574c744aaa838e1b7ef1304bff
I20260923 23:05:42.620601 12164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: loading cluster ID for follower catalog manager: success
I20260923 23:05:42.621706 12164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: acquiring CA information for follower catalog manager: success
I20260923 23:05:42.622037 12164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 23:05:42.689337 12101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:42.689595 12101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:42.689646 12101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:42.690948 12101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:42.691009 12101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:42.691033 12101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:42.691063 12101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:42.691089 12101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:42.692951 12101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:38663,127.5.74.61:38741,127.5.74.60:40011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.60.keytab
--principal=oryx/127.5.74.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.60:40011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.74.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:42.693251 12101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:42.693508 12101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:42.699286 12170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.699296 12169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.699280 12172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:42.700342 12101 server_base.cc:1034] running on GCE node
I20260923 23:05:42.700639 12101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:42.701210 12101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:42.702534 12101 hybrid_clock.cc:648] HybridClock initialized: now 1790204742702524 us; error 61 us; skew 500 ppm
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:42.705605 12101 init.cc:377] Logged in from keytab as oryx/127.5.74.60@KRBTEST.COM (short username oryx)
I20260923 23:05:42.706146 12101 webserver.cc:460] Webserver started at http://127.5.74.60:45551/ using document root <none> and password file <none>
I20260923 23:05:42.706405 12101 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:42.706475 12101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:42.706627 12101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:42.707875 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data/instance:
uuid: "d123501b5751435baa822d815d2d5b86"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "7d4ef2950284539061694291791dcccb"
server_key_iv: "07e8a33a227785d80a1374a63a0034e1"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.708375 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal/instance:
uuid: "d123501b5751435baa822d815d2d5b86"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "7d4ef2950284539061694291791dcccb"
server_key_iv: "07e8a33a227785d80a1374a63a0034e1"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.710340 12101 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260923 23:05:42.711441 12179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.711763 12101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:42.711892 12101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal
uuid: "d123501b5751435baa822d815d2d5b86"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "7d4ef2950284539061694291791dcccb"
server_key_iv: "07e8a33a227785d80a1374a63a0034e1"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.711992 12101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:42.746021 12101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:42.748543 12101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:42.748745 12101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:42.754153 12101 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.60:40011
I20260923 23:05:42.754215 12231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.60:40011 every 8 connection(s)
I20260923 23:05:42.754594 12101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
I20260923 23:05:42.755708 12232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:42.757563 12232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:05:42.764134 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 12101
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.60@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:42.768141 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.758534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.5.74.62:38663)
0923 23:05:42.758590 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.758592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.758810 (+ 218us) server_negotiation.cc:407] Connection header received
0923 23:05:42.759381 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.759383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.759395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.759467 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.760313 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.760825 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.762181 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.762496 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.762567 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.765217 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.765218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.765219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.765227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.766731 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.767156 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.767157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.767158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.767219 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.767487 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.767488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.767489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.767603 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.767681 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.767966 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.768046 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:42.768392 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.758416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:59720)
0923 23:05:42.758638 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.758682 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.759271 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.759583 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.759587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.759893 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.760123 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.760127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.760955 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.760956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.762077 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.762080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.762600 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.762601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.762717 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.763319 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.763332 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.765112 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.766825 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.766832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.766835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.767075 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.767288 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.767289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.767289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.767412 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.767672 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.767678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.767829 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.767943 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.768106 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":140,"threads_started":1}
I20260923 23:05:42.768913 12232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.74.60@KRBTEST.COM
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.60@KRBTEST.COM for oryx/127.5.74.61@KRBTEST.COM
I20260923 23:05:42.774204 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.764711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.5.74.60:40011)
0923 23:05:42.764902 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.764906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.764923 (+ 17us) server_negotiation.cc:407] Connection header received
0923 23:05:42.764945 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.764946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.764962 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.765046 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.765670 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.766372 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.767116 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.767427 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.767506 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.770562 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.770571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.770571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.770578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.772659 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.773050 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.773054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.773054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.773143 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.773358 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.773359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.773359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.773576 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.773800 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.773895 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.774062 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":93,"threads_started":1}
I20260923 23:05:42.776950 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.769196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.0.0.1:58898)
0923 23:05:42.769341 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.769353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.769425 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.769608 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.769609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.769773 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.769847 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.769849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.770919 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.770920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.771632 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.771635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.772237 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.772242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.772325 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.772768 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.772771 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.774121 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.775796 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.775797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.775797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.775990 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.776240 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.776240 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.776241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.776347 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.776653 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.776654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.776694 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.776783 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.776869 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260923 23:05:42.776962 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.769332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.5.74.61:38741)
0923 23:05:42.769473 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.769475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.769484 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:42.769506 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.769507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.769525 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.769580 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.769941 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.770788 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.771761 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.772081 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.772185 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.774205 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.774206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.774206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.774213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.775701 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.776067 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.776068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.776068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.776121 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.776429 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.776430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.776431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.776560 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.776629 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.776808 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.776882 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260923 23:05:42.777586 12232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.60@KRBTEST.COM for oryx/127.5.74.60@KRBTEST.COM
I20260923 23:05:42.784368 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.777871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40011 (local address 127.0.0.1:50958)
0923 23:05:42.777922 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.777933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.777971 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.778077 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.778078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.778217 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.778248 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.778249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.778923 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.778923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.779399 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.779400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.779780 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.779781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.779827 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.780362 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.780367 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.781642 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.783281 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.783282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.783282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.783455 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.783663 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.783664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.783664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.783717 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.783984 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.783985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.784022 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.784086 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.784224 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 23:05:42.784368 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.777939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.5.74.60:40011)
0923 23:05:42.777986 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.777988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.777996 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:42.778024 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.778024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.778033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.778076 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.778337 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.778831 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.779479 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.779690 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.779777 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.781759 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.781760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.781760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.781767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.783164 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.783541 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.783542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.783543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.783598 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.783790 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.783790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.783791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.783917 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.783994 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.784107 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.784273 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":33}
I20260923 23:05:42.786087 12232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Bootstrap starting.
I20260923 23:05:42.786983 12232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:42.787271 12232 log.cc:824] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:42.788223 12232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: No bootstrap required, opened a new log
I20260923 23:05:42.789775 12232 raft_consensus.cc:348] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } }
I20260923 23:05:42.789937 12232 raft_consensus.cc:374] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:42.789985 12232 raft_consensus.cc:729] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d123501b5751435baa822d815d2d5b86, State: Initialized, Role: FOLLOWER
I20260923 23:05:42.790149 12232 consensus_queue.cc:260] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } }
I20260923 23:05:42.790488 12232 sys_catalog.cc:564] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:42.790453 12238 sys_catalog.cc:455] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } } }
I20260923 23:05:42.790627 12238 sys_catalog.cc:458] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:05:42.792642 12238 raft_consensus.cc:482] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:42.792873 12238 raft_consensus.cc:504] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } }
W20260923 23:05:42.793023 12249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:05:42.793076 12249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:05:42.793102 12238 leader_election.cc:288] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663), 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61:38741)
I20260923 23:05:42.793357 12124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d123501b5751435baa822d815d2d5b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efbc7de087342ba8b39f238f8c23a37" is_pre_election: true
I20260923 23:05:42.793466 12124 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 LEARNER]: Handling vote request from an unknown peer d123501b5751435baa822d815d2d5b86
I20260923 23:05:42.793509 12124 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate d123501b5751435baa822d815d2d5b86 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260923 23:05:42.793519 11524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d123501b5751435baa822d815d2d5b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" is_pre_election: true
I20260923 23:05:42.793939 12182 leader_election.cc:302] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d123501b5751435baa822d815d2d5b86; no voters: 3efbc7de087342ba8b39f238f8c23a37, fb622f88b87c4f2c8c62912ca7a019e7
I20260923 23:05:42.794092 12238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:42.796514 12238 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 23:05:42.797078 12101 master_runner.cc:428] Detected that this master d123501b5751435baa822d815d2d5b86 is joining an existing cluster
I20260923 23:05:42.797132 12101 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.74.60:40011
I20260923 23:05:42.806434 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.798375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.0.0.1:58904)
0923 23:05:42.798532 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.798550 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.798604 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.798738 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.798739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.798922 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.798984 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.798985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.799984 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.799985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.801271 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.801273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.801659 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.801660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.801722 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.802208 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.802213 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.802542 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.805180 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.805181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.805181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.805465 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.805700 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.805701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.805702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.805771 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.806066 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.806067 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.806109 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.806167 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.806281 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260923 23:05:42.807013 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.798463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.5.74.61:38741)
0923 23:05:42.798511 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.798513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.798611 (+ 98us) server_negotiation.cc:407] Connection header received
0923 23:05:42.798662 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.798663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.798674 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.798737 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.799061 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.799856 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.801342 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.801560 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.801647 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.802654 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.802656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.802656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.802663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.804940 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.805558 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.805560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.805560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.805629 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.805839 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.805840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.805840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.805977 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.806040 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.806817 (+ 777us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.806919 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:05:42.808602 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.800584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59726 (local address 127.5.74.62:38663)
0923 23:05:42.800640 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.800642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.800789 (+ 147us) server_negotiation.cc:407] Connection header received
0923 23:05:42.800824 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.800826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.800837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.800901 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.801298 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.802047 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.803088 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.803298 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.803355 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.805241 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.805244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.805244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.805252 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.807294 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.807722 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.807723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.807724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.807797 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.808012 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.808013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.808014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.808134 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.808270 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.808432 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.808508 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:05:42.808620 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.800381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:59726)
0923 23:05:42.800695 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.800705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.800775 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.800919 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.800920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.801138 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.801212 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.801214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.802145 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.802146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.803012 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.803015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.803375 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.803376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.803440 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.804506 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.804511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.805107 (+ 596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.807407 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.807408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.807409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.807633 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.807885 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.807886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.807886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.807955 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.808261 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.808262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.808305 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.808392 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.808510 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":78,"threads_started":1}
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204741, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.62@KRBTEST.COM for oryx/127.5.74.60@KRBTEST.COM
I20260923 23:05:42.825637 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 11576
I20260923 23:05:42.828740 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.800590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40011 (local address 127.0.0.1:50974)
0923 23:05:42.804270 (+ 3680us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.804282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.804362 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.804651 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.804652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.805240 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.805308 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.805310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.806795 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.806795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.807861 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.807864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.808763 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.808764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.808840 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.809481 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.809487 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.824552 (+ 15065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.827426 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.827427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.827427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.827626 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.827893 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.827893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.827894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.827965 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.828332 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.828333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.828377 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.828437 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.828557 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3660,"thread_start_us":62,"threads_started":1}
I20260923 23:05:42.829599 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.800597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.5.74.60:40011)
0923 23:05:42.800640 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.800642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.804338 (+ 3696us) server_negotiation.cc:407] Connection header received
0923 23:05:42.804522 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.804524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.804537 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.804570 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.805420 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.806708 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.807960 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.808320 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.808416 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.825121 (+ 16705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.825123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.825124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.825132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.827308 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.827755 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.827756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.827756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.827815 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.828051 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.828052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.828052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.828244 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.828311 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.829410 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.829505 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:05:42.830164 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.811100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40011 (local address 127.0.0.1:50982)
0923 23:05:42.811155 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.811167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.811212 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.811521 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.811522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.811696 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.811744 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.811746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.812572 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.812574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.813112 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.813114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.813549 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.813550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.813632 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.814149 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.814154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.815556 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.828899 (+ 13343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.828902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.828903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.829153 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.829517 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.829518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.829519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.829589 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.829889 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.829891 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.829940 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.829987 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.830099 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 23:05:42.830611 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.811160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.5.74.60:40011)
0923 23:05:42.811358 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.811361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.811368 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:05:42.811412 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.811412 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.811437 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.811518 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.811829 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.812484 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.813223 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.813452 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.813548 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.815668 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.815670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.815670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.815679 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.827966 (+ 12287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.829360 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.829362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.829362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.829428 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.829663 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.829664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.829664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.829808 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.830406 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.830447 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.830530 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":117,"threads_started":1}
I20260923 23:05:42.831136 11514 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.74.60:40011: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d123501b5751435baa822d815d2d5b86"
member_type: NON_VOTER
last_known_addr {
host: "127.5.74.60"
port: 40011
}
attrs {
promote: true
}
}
dest_uuid: "fb622f88b87c4f2c8c62912ca7a019e7"
cas_config_opid_index: 5
I20260923 23:05:42.831430 11514 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } }
I20260923 23:05:42.831905 12072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d123501b5751435baa822d815d2d5b86 (127.5.74.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } } }
I20260923 23:05:42.832260 12206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Refusing update from remote peer fb622f88b87c4f2c8c62912ca7a019e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 23:05:42.833099 12072 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.74.60:40011
I20260923 23:05:42.833141 12259 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:42.833181 12072 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } } }
I20260923 23:05:42.833237 12072 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:42.834499 12206 raft_consensus.cc:858] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } }
W20260923 23:05:42.834575 12206 raft_consensus.cc:1565] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } }
I20260923 23:05:42.834765 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:46765
--local_ip_for_outbound_sockets=127.5.74.1
--tserver_master_addrs=127.5.74.62:38663
--webserver_port=35005
--webserver_interface=127.5.74.1
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab
--principal=oryx/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.74.62:38663,127.5.74.61:38741
--tserver_master_addrs=127.5.74.62:38663,127.5.74.61:38741,127.5.74.60:40011 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:05:42.835141 12206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } }
I20260923 23:05:42.835959 12260 mvcc.cc:205] Tried to move back new op lower bound from 7332678622171738112 to 7332678622085079040. Current Snapshot: MvccSnapshot[applied={T|T < 7332678622095675392}]
I20260923 23:05:42.836565 12238 sys_catalog.cc:455] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb622f88b87c4f2c8c62912ca7a019e7. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } }
I20260923 23:05:42.836649 12238 sys_catalog.cc:458] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 23:05:42.836683 12238 sys_catalog.cc:455] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } }
I20260923 23:05:42.836725 12238 sys_catalog.cc:458] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 23:05:42.836743 12262 sys_catalog.cc:455] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } }
I20260923 23:05:42.836912 12262 sys_catalog.cc:458] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 23:05:42.837347 12073 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } } }
I20260923 23:05:42.837467 12073 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:42.837751 12206 raft_consensus.cc:1467] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260923 23:05:42.837972 12206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d123501b5751435baa822d815d2d5b86 (127.5.74.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } } }
I20260923 23:05:42.838439 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.831919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.0.0.1:58914)
0923 23:05:42.832102 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.832113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.832200 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.832560 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.832561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.832563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:42.832609 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.832610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.833709 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.833711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.834860 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.834863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.838127 (+ 3264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.838129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.838209 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.838254 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.838297 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.838356 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260923 23:05:42.838886 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.832027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.5.74.61:38741)
0923 23:05:42.832072 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.832074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.832422 (+ 348us) server_negotiation.cc:407] Connection header received
0923 23:05:42.832451 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.832452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.832478 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.832546 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:42.832693 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.833574 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.836398 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.837991 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.838634 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.838744 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.838768 (+ 24us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.838815 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:05:42.839104 12124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 LEARNER]: Refusing update from remote peer fb622f88b87c4f2c8c62912ca7a019e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 23:05:42.839403 12072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 23:05:42.839413 12262 sys_catalog.cc:455] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } } }
I20260923 23:05:42.839715 12262 sys_catalog.cc:458] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: This master's current role is: LEARNER
I20260923 23:05:42.839756 12259 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: attempting to promote NON_VOTER d123501b5751435baa822d815d2d5b86 to VOTER
I20260923 23:05:42.840339 12124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d123501b5751435baa822d815d2d5b86 (127.5.74.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } } }
I20260923 23:05:42.840438 12259 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:42.841501 12206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 LEARNER]: Refusing update from remote peer fb622f88b87c4f2c8c62912ca7a019e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 23:05:42.841603 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb622f88b87c4f2c8c62912ca7a019e7. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } } }
I20260923 23:05:42.841699 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: This master's current role is: LEARNER
I20260923 23:05:42.841740 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } } }
I20260923 23:05:42.841789 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: This master's current role is: LEARNER
I20260923 23:05:42.841840 12259 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 23:05:42.842563 12101 master.cc:537] Master@127.5.74.60:40011 shutting down...
I20260923 23:05:42.843254 12072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d123501b5751435baa822d815d2d5b86 (127.5.74.60) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
W20260923 23:05:42.843725 11500 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 -> Peer d123501b5751435baa822d815d2d5b86 (127.5.74.60:40011): Couldn't send request to peer d123501b5751435baa822d815d2d5b86. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 23:05:42.845408 12072 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.845534 12072 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:42.845597 12072 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.845646 12072 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:42.847563 12123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 LEARNER]: Refusing update from remote peer fb622f88b87c4f2c8c62912ca7a019e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 23:05:42.847729 12101 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:42.847885 12101 pending_rounds.cc:62] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Trying to abort 1 pending ops.
I20260923 23:05:42.847965 12259 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 23:05:42.847936 12101 pending_rounds.cc:68] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7332678626673995776 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "00000000000000000000000000000000" old_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: NON_VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } } }
I20260923 23:05:42.848075 12101 raft_consensus.cc:2894] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20260923 23:05:42.848112 12101 raft_consensus.cc:2866] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260923 23:05:42.848140 12101 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 LEARNER]: Raft consensus is shut down!
I20260923 23:05:42.848217 12101 tablet_replica.cc:330] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: stopping tablet replica
I20260923 23:05:42.848449 12123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d123501b5751435baa822d815d2d5b86 (127.5.74.60) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.849606 12269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: NON_VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: true } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.849793 12269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: This master's current role is: LEARNER
I20260923 23:05:42.850121 12259 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: attempting to promote NON_VOTER 3efbc7de087342ba8b39f238f8c23a37 to VOTER
I20260923 23:05:42.850270 12259 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:42.850912 12123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 LEARNER]: Refusing update from remote peer fb622f88b87c4f2c8c62912ca7a019e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 23:05:42.851166 12072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260923 23:05:42.851464 11500 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 -> Peer d123501b5751435baa822d815d2d5b86 (127.5.74.60:40011): Couldn't send request to peer d123501b5751435baa822d815d2d5b86. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 23:05:42.851871 12072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.852877 12072 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.852962 12072 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:42.853025 12072 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.853080 12072 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:42.853361 12123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.854578 12269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.854687 12269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:05:42.855468 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.855299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40011 (local address 127.0.0.1:50986)
0923 23:05:42.855385 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.855430 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 23:05:42.863936 12101 master.cc:552] Master@127.5.74.60:40011 shutdown complete.
I20260923 23:05:42.864008 12101 master_runner.cc:305] Clearing existing system tablet
I20260923 23:05:42.864286 12101 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:42.864403 12101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:42.865477 12101 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:42.866081 12271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.866194 12101 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:42.866238 12101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:05:42.866258 12101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal
uuid: "d123501b5751435baa822d815d2d5b86"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "7d4ef2950284539061694291791dcccb"
server_key_iv: "07e8a33a227785d80a1374a63a0034e1"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.866989 12101 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:42.868695 12101 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 23:05:42.868770 12101 log.cc:1196] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 23:05:42.869021 12101 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Deleting consensus metadata
I20260923 23:05:42.869138 12101 master_runner.cc:315] Copying system tablet from 127.5.74.62:38663
I20260923 23:05:42.869733 12101 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: tablet copy: Beginning tablet copy session from remote peer at address 127.5.74.62:38663
I20260923 23:05:42.878888 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.870088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:59738)
0923 23:05:42.870268 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:42.870280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:42.870363 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:42.870502 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:42.870503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:42.870704 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:42.870762 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.870763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.871778 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.871780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.872866 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:42.872871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.873390 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.873392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:42.873448 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.874060 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:42.874065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:42.874634 (+ 569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:42.877413 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.877415 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.877416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.877833 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.878101 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:42.878102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:42.878103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:42.878175 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:42.878487 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:42.878488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:42.878543 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 23:05:42.878616 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 23:05:42.878738 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260923 23:05:42.878882 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:42.870165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.5.74.62:38663)
0923 23:05:42.870216 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:42.870218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:42.870370 (+ 152us) server_negotiation.cc:407] Connection header received
0923 23:05:42.870416 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:42.870418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:42.870430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:42.870496 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:42.870862 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.871674 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.872976 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:42.873283 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:42.873378 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:42.874885 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:42.874887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:42.874887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:42.874895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:42.876948 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.877938 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.877939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.877940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.878018 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:42.878251 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:42.878252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:42.878252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:42.878413 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:42.878493 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:42.878677 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 23:05:42.878771 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:05:42.880705 11534 tablet_copy_service.cc:140] P fb622f88b87c4f2c8c62912ca7a019e7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d123501b5751435baa822d815d2d5b86 ({username='oryx', principal='oryx/127.5.74.60@KRBTEST.COM'} at 127.0.0.1:59738)
I20260923 23:05:42.880808 11534 tablet_copy_service.cc:161] P fb622f88b87c4f2c8c62912ca7a019e7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d123501b5751435baa822d815d2d5b86 at {username='oryx', principal='oryx/127.5.74.60@KRBTEST.COM'} at 127.0.0.1:59738: session id = d123501b5751435baa822d815d2d5b86-00000000000000000000000000000000
I20260923 23:05:42.881780 11534 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 23:05:42.882560 12101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 23:05:42.884977 12101 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: tablet copy: Starting download of 0 data blocks...
I20260923 23:05:42.885249 12101 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: tablet copy: Starting download of 1 WAL segments...
I20260923 23:05:42.886657 12101 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 23:05:42.888376 11534 tablet_copy_service.cc:342] P fb622f88b87c4f2c8c62912ca7a019e7: Request end of tablet copy session d123501b5751435baa822d815d2d5b86-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.74.60@KRBTEST.COM'} at 127.0.0.1:59738
I20260923 23:05:42.888465 11534 tablet_copy_service.cc:433] P fb622f88b87c4f2c8c62912ca7a019e7: ending tablet copy session d123501b5751435baa822d815d2d5b86-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d123501b5751435baa822d815d2d5b86
I20260923 23:05:42.891916 12101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:42.893664 12278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.893664 12281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:42.896334 12101 server_base.cc:1034] running on GCE node
W20260923 23:05:42.896356 12279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:42.896552 12101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:42.896750 12101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:42.897893 12101 hybrid_clock.cc:648] HybridClock initialized: now 1790204742897877 us; error 33 us; skew 500 ppm
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:42.899600 12101 init.cc:377] Logged in from keytab as oryx/127.5.74.60@KRBTEST.COM (short username oryx)
I20260923 23:05:42.899930 12101 webserver.cc:460] Webserver started at http://127.5.74.60:36905/ using document root <none> and password file <none>
I20260923 23:05:42.900065 12101 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:42.900110 12101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:42.901073 12101 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:42.901822 12288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.902002 12101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:42.902066 12101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal
uuid: "d123501b5751435baa822d815d2d5b86"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "7d4ef2950284539061694291791dcccb"
server_key_iv: "07e8a33a227785d80a1374a63a0034e1"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.902135 12101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:42.918282 12101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:42.920819 12101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:42.925587 12101 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.60:40011
I20260923 23:05:42.925633 12340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.60:40011 every 8 connection(s)
I20260923 23:05:42.926067 12101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
I20260923 23:05:42.927227 12341 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:05:42.927865 12341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Bootstrap starting.
I20260923 23:05:42.931113 12341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:05:42.931442 12341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Bootstrap complete.
I20260923 23:05:42.931708 12341 raft_consensus.cc:348] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:42.931823 12341 raft_consensus.cc:729] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d123501b5751435baa822d815d2d5b86, State: Initialized, Role: FOLLOWER
I20260923 23:05:42.931927 12341 consensus_queue.cc:260] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:42.932145 12341 sys_catalog.cc:564] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:42.932144 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:42.932623 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:05:42.934839 12355 catalog_manager.cc:1259] Loaded cluster ID: 392d22574c744aaa838e1b7ef1304bff
I20260923 23:05:42.934897 12355 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: loading cluster ID for follower catalog manager: success
I20260923 23:05:42.936197 12355 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: acquiring CA information for follower catalog manager: success
I20260923 23:05:42.936542 12355 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 23:05:42.951349 12266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:42.951625 12266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:42.951671 12266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:42.953107 12266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:42.953181 12266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:42.953238 12266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:42.954912 12266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab
--principal=oryx/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:46765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=35005
--tserver_master_addrs=127.5.74.62:38663,127.5.74.61:38741,127.5.74.60:40011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:42.955202 12266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:42.955467 12266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:42.961287 12360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:42.961326 12363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:42.961565 12266 server_base.cc:1034] running on GCE node
W20260923 23:05:42.961340 12361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:42.961938 12266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:42.962285 12266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:42.963510 12266 hybrid_clock.cc:648] HybridClock initialized: now 1790204742963484 us; error 41 us; skew 500 ppm
Sep 23 23:05:42 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:42.966652 12266 init.cc:377] Logged in from keytab as oryx/127.5.74.1@KRBTEST.COM (short username oryx)
I20260923 23:05:42.967214 12266 webserver.cc:460] Webserver started at http://127.5.74.1:35005/ using document root <none> and password file <none>
I20260923 23:05:42.967506 12266 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:42.967600 12266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:42.969405 12266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:42.970250 12370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.970597 12266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:42.970731 12266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "61144ca035074cf6b49ac17c874d9d42"
format_stamp: "Formatted at 2026-09-23 23:05:41 on dist-test-slave-j4c5"
server_key: "ab1d6ec7639fd1e6eb52d6e3510689c6"
server_key_iv: "84c3d5dcadb760600dcd22173c51a991"
server_key_version: "encryptionkey@0"
I20260923 23:05:42.971261 12266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:42.988117 12266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:42.990806 12266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:42.991011 12266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:42.991307 12266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:42.991634 12266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:42.991688 12266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.991730 12266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:42.991761 12266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:42.998992 12266 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:46765
I20260923 23:05:42.999079 12483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:46765 every 8 connection(s)
I20260923 23:05:42.999464 12266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:43.006891 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 12266
I20260923 23:05:43.007120 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 11710
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.60@KRBTEST.COM
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.61@KRBTEST.COM
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204742, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:43.014806 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.000706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:43435 (local address 127.5.74.62:38663)
0923 23:05:43.000752 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.000755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.001625 (+ 870us) server_negotiation.cc:407] Connection header received
0923 23:05:43.001663 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.001665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.001680 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.001740 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.002855 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.004040 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.006012 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.006500 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.006591 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.010524 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.010526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.010527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.010535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.013001 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013689 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.013690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.013690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.013749 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.014007 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.014008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.014008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.014137 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.014178 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.014586 (+ 408us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.014674 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:05:43.014809 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.000652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:34245 (local address 127.5.74.61:38741)
0923 23:05:43.000697 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.000699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.000852 (+ 153us) server_negotiation.cc:407] Connection header received
0923 23:05:43.001469 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.001470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.001479 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.001575 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.002750 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.003494 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.005653 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.005930 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.005972 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.009864 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.009866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.009866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.009876 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.012818 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013584 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.013585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.013585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.013654 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013967 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.013968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.013968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.014116 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.014165 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.014595 (+ 430us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.014687 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:05:43.015177 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.000627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.1:43435)
0923 23:05:43.001388 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.001541 (+ 153us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.001607 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.001756 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.001757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.002242 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:43.002635 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.002641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.004202 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.004204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.005744 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.005745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.006592 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.006593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.006638 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.008221 (+ 1583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:43.008227 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:43.010405 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:43.013110 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.013112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.013605 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.013821 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.013822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.013880 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.014323 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:43.014324 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:43.014411 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.014919 (+ 508us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.015064 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":107,"thread_start_us":63,"threads_started":1}
I20260923 23:05:43.015183 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.000272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40011 (local address 127.5.74.1:52519)
0923 23:05:43.000616 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.000763 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.001246 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.001572 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.001576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.002261 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:43.002635 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.002642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.004526 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.004527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.005196 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.005198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.006154 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.006155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.006222 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.006976 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:43.006988 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:43.010405 (+ 3417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:43.013110 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.013112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.013511 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.013731 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.013732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.013879 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.014281 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:43.014282 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:43.014411 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.014535 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.014707 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":128,"thread_start_us":77,"threads_started":1}
I20260923 23:05:43.015484 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.000448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.5.74.1:34245)
0923 23:05:43.000747 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.000766 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.001335 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.001631 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.001632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.002243 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:43.002635 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.002641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.003655 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.003656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.005518 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.005519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.006053 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.006054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.006204 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.007013 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:43.007018 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:43.009726 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:43.012937 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.012950 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.012953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.013475 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.013751 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.013753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.013879 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.014219 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:43.014226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:43.014413 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.014958 (+ 545us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.015161 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":158,"thread_start_us":102,"threads_started":1}
I20260923 23:05:43.015767 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.000264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:52519 (local address 127.5.74.60:40011)
0923 23:05:43.000482 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.000485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.000874 (+ 389us) server_negotiation.cc:407] Connection header received
0923 23:05:43.001361 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.001363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.001377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.001486 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.002927 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.004408 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.005555 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.005813 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.006167 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.010516 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.010518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.010518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.010527 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.012997 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013598 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.013599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.013599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.013663 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.013948 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.013948 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.013949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.014100 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.014164 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.015541 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.015650 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":64,"threads_started":1}
I20260923 23:05:43.015940 12486 heartbeater.cc:347] Connected to a master server at 127.5.74.61:38741
I20260923 23:05:43.016041 12486 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:43.016278 12486 heartbeater.cc:511] Master 127.5.74.61:38741 requested a full tablet report, sending...
I20260923 23:05:43.016924 12114 ts_manager.cc:194] Registered new tserver with Master: 61144ca035074cf6b49ac17c874d9d42 (127.5.74.1:46765)
I20260923 23:05:43.017751 12485 heartbeater.cc:347] Connected to a master server at 127.5.74.62:38663
I20260923 23:05:43.017817 12485 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:43.017855 12484 heartbeater.cc:347] Connected to a master server at 127.5.74.60:40011
I20260923 23:05:43.017897 12485 heartbeater.cc:511] Master 127.5.74.62:38663 requested a full tablet report, sending...
I20260923 23:05:43.017899 12484 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:43.018038 12484 heartbeater.cc:511] Master 127.5.74.60:40011 requested a full tablet report, sending...
I20260923 23:05:43.018226 11514 ts_manager.cc:194] Re-registered known tserver with Master: 61144ca035074cf6b49ac17c874d9d42 (127.5.74.1:46765)
I20260923 23:05:43.018452 12305 ts_manager.cc:194] Registered new tserver with Master: 61144ca035074cf6b49ac17c874d9d42 (127.5.74.1:46765)
I20260923 23:05:43.018712 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:43089
--local_ip_for_outbound_sockets=127.5.74.2
--tserver_master_addrs=127.5.74.62:38663
--webserver_port=41311
--webserver_interface=127.5.74.2
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab
--principal=oryx/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.74.62:38663,127.5.74.61:38741
--tserver_master_addrs=127.5.74.62:38663,127.5.74.61:38741,127.5.74.60:40011 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:05:43.018827 11514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:43435
W20260923 23:05:43.111591 12492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:43.111838 12492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:43.111868 12492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:43.113137 12492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:43.113211 12492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:43.113250 12492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:43.114781 12492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab
--principal=oryx/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:43089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=41311
--tserver_master_addrs=127.5.74.62:38663,127.5.74.61:38741,127.5.74.60:40011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:43.115000 12492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:43.115195 12492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:43.120981 12498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:43.120906 12497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:43.121280 12500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:43.121717 12492 server_base.cc:1034] running on GCE node
I20260923 23:05:43.122083 12492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:43.122402 12492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:43.125711 12492 hybrid_clock.cc:648] HybridClock initialized: now 1790204743125680 us; error 39 us; skew 500 ppm
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204743, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:43.129325 12492 init.cc:377] Logged in from keytab as oryx/127.5.74.2@KRBTEST.COM (short username oryx)
I20260923 23:05:43.130079 12492 webserver.cc:460] Webserver started at http://127.5.74.2:41311/ using document root <none> and password file <none>
I20260923 23:05:43.130347 12492 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:43.130438 12492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:43.132685 12492 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:43.133461 12507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:43.133731 12492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:43.133848 12492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "1bc9bd0f474f425895102fae742d8082"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "5cb3028f53455b2cfa4aac0c86d91d9c"
server_key_iv: "42255bc999f16f92dc4cf0f81a1d1377"
server_key_version: "encryptionkey@0"
I20260923 23:05:43.134220 12492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:43.155566 12492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:43.158340 12492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:43.158510 12492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:43.158784 12492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:43.159106 12492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:43.159140 12492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:43.159163 12492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:43.159212 12492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:43.166356 12492 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:43089
I20260923 23:05:43.166393 12620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:43089 every 8 connection(s)
I20260923 23:05:43.166806 12492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204743, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.60@KRBTEST.COM
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204743, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:43.175220 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 12492
I20260923 23:05:43.175400 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 11844
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204743, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.61@KRBTEST.COM
I20260923 23:05:43.178314 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.167976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:42587 (local address 127.5.74.60:40011)
0923 23:05:43.168014 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.168016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.168277 (+ 261us) server_negotiation.cc:407] Connection header received
0923 23:05:43.168916 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.168917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.168933 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.168995 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.170131 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.170646 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.171714 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.171929 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.172017 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.174811 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.174813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.174813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.174823 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.176654 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.177169 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.177170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.177171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.177229 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.177505 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.177506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.177506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.177654 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.177701 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.178098 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.178193 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:05:43.178710 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.167620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:34771 (local address 127.5.74.62:38663)
0923 23:05:43.167684 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.167687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.168048 (+ 361us) server_negotiation.cc:407] Connection header received
0923 23:05:43.168792 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.168795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.168816 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.168891 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.170170 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.170950 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.172139 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.172472 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.172535 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.175224 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.175226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.175226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.175289 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.177383 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.177794 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.177796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.177796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.177858 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.178068 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.178069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.178069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.178213 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.178274 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.178489 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.178582 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:05:43.178860 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.167891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40011 (local address 127.5.74.2:42587)
0923 23:05:43.168207 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.168215 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.168739 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.169382 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.169384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.169738 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:43.170002 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.170008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.170759 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.170760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.171596 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.171598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.172025 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.172026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.172214 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.172918 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:43.172931 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:43.174673 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:43.176749 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.176757 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.176761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.177081 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.177299 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.177300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.177300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.177434 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.177751 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:43.177758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:43.177943 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.178401 (+ 458us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.178591 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":55,"threads_started":1}
I20260923 23:05:43.179204 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.167595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:34771)
0923 23:05:43.167994 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.168003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.168604 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.169105 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.169112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.169872 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:43.170007 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.170011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.171067 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.171067 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.172008 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.172010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.172555 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.172556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.172640 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.173066 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:43.173070 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:43.175131 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:43.177481 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.177482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.177483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.177704 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.177931 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.177932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.177932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.177999 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.178323 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:43.178324 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:43.178374 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.178922 (+ 548us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.179048 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":44,"thread_start_us":88,"threads_started":1}
I20260923 23:05:43.179728 12622 heartbeater.cc:347] Connected to a master server at 127.5.74.62:38663
I20260923 23:05:43.179838 12622 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:43.180037 12622 heartbeater.cc:511] Master 127.5.74.62:38663 requested a full tablet report, sending...
I20260923 23:05:43.180505 11514 ts_manager.cc:194] Re-registered known tserver with Master: 1bc9bd0f474f425895102fae742d8082 (127.5.74.2:43089)
I20260923 23:05:43.181033 11514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:34771
I20260923 23:05:43.181239 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.167682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:42789 (local address 127.5.74.61:38741)
0923 23:05:43.167724 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.167726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.168051 (+ 325us) server_negotiation.cc:407] Connection header received
0923 23:05:43.168800 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.168802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.168848 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.168952 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.170167 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.170947 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.172103 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.172375 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.172414 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.178374 (+ 5960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.178376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.178376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.178385 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.180005 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.180421 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.180422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.180422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.180481 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.180687 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.180688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.180688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.180828 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.180876 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.181052 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.181131 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:05:43.181633 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.167595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.5.74.2:42789)
0923 23:05:43.167922 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.167970 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.168604 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.169130 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.169131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.169872 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:43.170007 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.170011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.171067 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.171068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.171981 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.171984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.172556 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.172556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.172640 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.173214 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:43.173218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:43.178250 (+ 5032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:43.180104 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.180105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.180105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.180337 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.180553 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.180553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.180553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.180606 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.180908 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:43.180908 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:43.180949 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.181357 (+ 408us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.181485 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":26,"thread_start_us":92,"threads_started":1}
I20260923 23:05:43.181943 12623 heartbeater.cc:347] Connected to a master server at 127.5.74.61:38741
I20260923 23:05:43.181984 12623 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:43.182209 12623 heartbeater.cc:511] Master 127.5.74.61:38741 requested a full tablet report, sending...
I20260923 23:05:43.182253 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:35743
--local_ip_for_outbound_sockets=127.5.74.3
--tserver_master_addrs=127.5.74.62:38663
--webserver_port=41585
--webserver_interface=127.5.74.3
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab
--principal=oryx/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.74.62:38663,127.5.74.61:38741
--tserver_master_addrs=127.5.74.62:38663,127.5.74.61:38741,127.5.74.60:40011 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:05:43.182377 12621 heartbeater.cc:347] Connected to a master server at 127.5.74.60:40011
I20260923 23:05:43.182415 12621 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:43.182473 12621 heartbeater.cc:511] Master 127.5.74.60:40011 requested a full tablet report, sending...
I20260923 23:05:43.182538 12114 ts_manager.cc:194] Registered new tserver with Master: 1bc9bd0f474f425895102fae742d8082 (127.5.74.2:43089)
I20260923 23:05:43.182729 12305 ts_manager.cc:194] Registered new tserver with Master: 1bc9bd0f474f425895102fae742d8082 (127.5.74.2:43089)
W20260923 23:05:43.270673 12628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:43.270964 12628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:43.271009 12628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:43.272316 12628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:43.272393 12628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:43.272444 12628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:43.274246 12628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab
--principal=oryx/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:35743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=41585
--tserver_master_addrs=127.5.74.62:38663,127.5.74.61:38741,127.5.74.60:40011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:43.274571 12628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:43.274791 12628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:43.279996 12633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:43.279999 12636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:43.279997 12634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:43.280357 12628 server_base.cc:1034] running on GCE node
I20260923 23:05:43.280646 12628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:43.280870 12628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:43.282071 12628 hybrid_clock.cc:648] HybridClock initialized: now 1790204743282048 us; error 30 us; skew 500 ppm
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204743, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:43.285055 12628 init.cc:377] Logged in from keytab as oryx/127.5.74.3@KRBTEST.COM (short username oryx)
I20260923 23:05:43.285583 12628 webserver.cc:460] Webserver started at http://127.5.74.3:41585/ using document root <none> and password file <none>
I20260923 23:05:43.285939 12628 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:43.286056 12628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:43.287739 12628 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 23:05:43.288494 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:43.288718 12628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:43.288823 12628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "cd81e0e2f06a45d0beaae3377efa1dc2"
format_stamp: "Formatted at 2026-09-23 23:05:42 on dist-test-slave-j4c5"
server_key: "dc29ef22996ef59db663020b7ec86cea"
server_key_iv: "34ef84ed2f08934485ce0763051bdaa4"
server_key_version: "encryptionkey@0"
I20260923 23:05:43.289192 12628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:43.312510 12628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:43.315145 12628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:43.315351 12628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:43.315618 12628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:43.315958 12628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:43.316036 12628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:43.316082 12628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:43.316131 12628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:43.323728 12628 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:35743
I20260923 23:05:43.323766 12756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:35743 every 8 connection(s)
I20260923 23:05:43.324240 12628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:05:43.327932 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 12628
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204743, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.60@KRBTEST.COM
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204743, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
Sep 23 23:05:43 dist-test-slave-j4c5 krb5kdc[11469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204743, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.61@KRBTEST.COM
I20260923 23:05:43.336040 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.325145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:60653 (local address 127.5.74.60:40011)
0923 23:05:43.325199 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.325202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.325542 (+ 340us) server_negotiation.cc:407] Connection header received
0923 23:05:43.326190 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.326191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.326203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.326242 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.327658 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.328427 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.329436 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.329748 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.329791 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.332614 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.332616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.332616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.332625 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.334356 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.334890 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.334891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.334891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.334953 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.335247 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.335247 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.335248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.335378 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.335424 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.335840 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.335916 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:05:43.336928 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.325021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40011 (local address 127.5.74.3:60653)
0923 23:05:43.325221 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.325273 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.326083 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.326357 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.326357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.326804 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:43.326942 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.326948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.328536 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.328537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.329349 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.329350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.329871 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.329872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.329920 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.330298 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:43.330302 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:43.332435 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:43.334439 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.334448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.334451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.334794 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.335030 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.335031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.335031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.335167 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.335474 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:43.335481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:43.335652 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.336393 (+ 741us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.336627 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":4,"thread_start_us":105,"threads_started":1}
I20260923 23:05:43.337770 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.325410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:45685 (local address 127.5.74.61:38741)
0923 23:05:43.325454 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.325456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.325563 (+ 107us) server_negotiation.cc:407] Connection header received
0923 23:05:43.326177 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.326179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.326192 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.326254 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.328549 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.329258 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.330377 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.330665 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.330710 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.333809 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.333811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.333811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.333819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.336406 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.336799 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.336800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.336800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.336854 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.337067 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.337068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.337069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.337239 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.337338 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.337581 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.337668 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:05:43.338151 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.325232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.5.74.3:45685)
0923 23:05:43.325388 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.325401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.326058 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.326329 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.326334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.326740 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:43.326941 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.326945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.329365 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.329366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.330208 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.330211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.330798 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.330798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.330877 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.331356 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:43.331362 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:43.333416 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:43.336489 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.336490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.336490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.336715 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.336924 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.336925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.336925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.337005 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.337399 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:43.337400 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:43.337450 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.337895 (+ 445us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.338010 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260923 23:05:43.338600 12759 heartbeater.cc:347] Connected to a master server at 127.5.74.61:38741
I20260923 23:05:43.338655 12757 heartbeater.cc:347] Connected to a master server at 127.5.74.60:40011
I20260923 23:05:43.338709 12757 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:43.338927 12757 heartbeater.cc:511] Master 127.5.74.60:40011 requested a full tablet report, sending...
I20260923 23:05:43.338711 12759 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:43.339125 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.325412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:39267 (local address 127.5.74.62:38663)
0923 23:05:43.325454 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.325456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.325670 (+ 214us) server_negotiation.cc:407] Connection header received
0923 23:05:43.326190 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.326191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.326203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.326238 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.327051 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.327565 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.328906 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.329209 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.329249 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.334095 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.334097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.334097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.334105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.337400 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.337973 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.337974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.337975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.338038 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.338450 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.338451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.338452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.338596 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.338648 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.338905 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.339011 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:05:43.339147 12759 heartbeater.cc:511] Master 127.5.74.61:38741 requested a full tablet report, sending...
I20260923 23:05:43.339514 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.325355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.3:39267)
0923 23:05:43.325612 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.325621 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.326083 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.326331 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.326334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.326734 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:43.326942 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.326948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.327670 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.327671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.328806 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.328808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.329331 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.329332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.329479 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.330128 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:43.330137 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:43.333698 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:43.337637 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.337638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.337639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.337872 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.338120 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:43.338121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:43.338121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:43.338335 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:43.338701 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:43.338702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:43.338772 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.339249 (+ 477us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.339369 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":25,"thread_start_us":168,"threads_started":1}
I20260923 23:05:43.339517 12114 ts_manager.cc:194] Registered new tserver with Master: cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743)
I20260923 23:05:43.340174 12758 heartbeater.cc:347] Connected to a master server at 127.5.74.62:38663
I20260923 23:05:43.340232 12758 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:43.340306 12758 heartbeater.cc:511] Master 127.5.74.62:38663 requested a full tablet report, sending...
I20260923 23:05:43.340529 12305 ts_manager.cc:194] Registered new tserver with Master: cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743)
I20260923 23:05:43.340971 11514 ts_manager.cc:194] Re-registered known tserver with Master: cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743)
I20260923 23:05:43.341464 11514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:39267
I20260923 23:05:43.343438 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.330305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.5.74.61:38741)
0923 23:05:43.330446 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.330448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.332484 (+ 2036us) server_negotiation.cc:407] Connection header received
0923 23:05:43.332515 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.332516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.332531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.332597 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.333594 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.334342 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.335517 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.335840 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.335893 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.340071 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.340073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.340074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.340082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.342058 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.342474 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.342476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.342476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.342539 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.342774 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.342775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.342776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.342917 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.343019 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.343268 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.343352 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260923 23:05:43.344345 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.329417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.5.74.60:40011)
0923 23:05:43.330574 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.330577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.330774 (+ 197us) server_negotiation.cc:407] Connection header received
0923 23:05:43.330797 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.330798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.330811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.330881 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.335040 (+ 4159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.337451 (+ 2411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.338968 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.339334 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.339393 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.340817 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.340818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.340818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.340825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.342584 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.342982 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.342984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.342984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.343055 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.343266 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.343267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.343267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.343417 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.343621 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.344108 (+ 487us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.344240 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1137,"thread_start_us":49,"threads_started":1}
I20260923 23:05:43.347602 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.329049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.5.74.62:38663)
0923 23:05:43.330969 (+ 1920us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.330971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.330980 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:43.331003 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.331004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.331331 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.331414 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.332543 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.341649 (+ 9106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.343041 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.343376 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.343453 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.344874 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.344875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.344875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.344884 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.346154 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.346530 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.346531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.346531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.346593 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.346920 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.346921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.346921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.347050 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.347143 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.347362 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.347476 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1887,"thread_start_us":76,"threads_started":1}
I20260923 23:05:43.350083 11514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:43.351676 11514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:43.360770 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.356567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:35743 (local address 127.0.0.1:49970)
0923 23:05:43.356653 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.356665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.356714 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.357540 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.357541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.357544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:43.357601 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.357603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.358688 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.358689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.359618 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.359620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.360520 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.360521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.360582 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.360587 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.360618 (+ 31us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.360657 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":42,"threads_started":1}
I20260923 23:05:43.361351 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.357109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.5.74.3:35743)
0923 23:05:43.357245 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.357250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.357260 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:43.357302 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.357304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.357322 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.357418 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:43.357680 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.358609 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.359701 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.360425 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.360957 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.361048 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.361075 (+ 27us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.361114 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260923 23:05:43.361503 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.356991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.5.74.2:43089)
0923 23:05:43.357175 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.357180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.357191 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:43.357412 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.357419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.357448 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.357591 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:43.357862 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.359125 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.360508 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.361145 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.361232 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.361315 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.361389 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.361429 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260923 23:05:43.361507 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.356730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:43089 (local address 127.0.0.1:47158)
0923 23:05:43.356889 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.356900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.356953 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.357607 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.357609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.357613 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:43.357753 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.357756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.359250 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.359252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.360421 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.360423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.361232 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.361233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.361310 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.361314 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.361375 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.361416 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":113,"threads_started":1}
I20260923 23:05:43.362109 12691 tablet_service.cc:1467] Processing CreateTablet for tablet 902a7d4e4ae94de49fa2197a62fcfd0c (DEFAULT_TABLE table=test-table [id=711487112d56412fb245460d55d1c367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:43.362504 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.356713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:46765 (local address 127.0.0.1:44118)
0923 23:05:43.356827 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.356836 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.356876 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.357998 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.357999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.358002 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:43.358193 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.358195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.360301 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.360302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.361385 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.361387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.362272 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.362273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.362357 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.362361 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.362393 (+ 32us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.362431 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":86,"threads_started":1}
I20260923 23:05:43.362509 12691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902a7d4e4ae94de49fa2197a62fcfd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:43.362953 12555 tablet_service.cc:1467] Processing CreateTablet for tablet 902a7d4e4ae94de49fa2197a62fcfd0c (DEFAULT_TABLE table=test-table [id=711487112d56412fb245460d55d1c367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:43.363135 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.357034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.5.74.1:46765)
0923 23:05:43.357793 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.357799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.357811 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:05:43.357858 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.357860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.357877 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.357976 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:43.358304 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.360211 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.361491 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.362160 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.362681 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.362979 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.363012 (+ 33us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.363059 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":103,"threads_started":1}
I20260923 23:05:43.363296 12555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902a7d4e4ae94de49fa2197a62fcfd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:43.363736 12418 tablet_service.cc:1467] Processing CreateTablet for tablet 902a7d4e4ae94de49fa2197a62fcfd0c (DEFAULT_TABLE table=test-table [id=711487112d56412fb245460d55d1c367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:43.364045 12418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902a7d4e4ae94de49fa2197a62fcfd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:43.366254 12785 tablet_bootstrap.cc:492] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: Bootstrap starting.
I20260923 23:05:43.366564 12786 tablet_bootstrap.cc:492] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: Bootstrap starting.
I20260923 23:05:43.367053 12785 tablet_bootstrap.cc:654] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:43.367137 12786 tablet_bootstrap.cc:654] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:43.367507 12786 log.cc:824] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:43.367507 12785 log.cc:824] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:43.367769 12787 tablet_bootstrap.cc:492] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: Bootstrap starting.
I20260923 23:05:43.368512 12786 tablet_bootstrap.cc:492] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: No bootstrap required, opened a new log
I20260923 23:05:43.368512 12785 tablet_bootstrap.cc:492] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: No bootstrap required, opened a new log
I20260923 23:05:43.368546 12787 tablet_bootstrap.cc:654] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:43.368585 12785 ts_tablet_manager.cc:1397] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:43.368585 12786 ts_tablet_manager.cc:1397] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:43.368955 12787 log.cc:824] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:43.369637 12787 tablet_bootstrap.cc:492] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: No bootstrap required, opened a new log
I20260923 23:05:43.369720 12787 ts_tablet_manager.cc:1397] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:05:43.371042 12787 raft_consensus.cc:348] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:43.371212 12787 raft_consensus.cc:374] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:43.371265 12787 raft_consensus.cc:729] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61144ca035074cf6b49ac17c874d9d42, State: Initialized, Role: FOLLOWER
I20260923 23:05:43.371416 12787 consensus_queue.cc:260] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:43.371490 12785 raft_consensus.cc:348] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:43.371670 12785 raft_consensus.cc:374] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:43.371713 12785 raft_consensus.cc:729] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd81e0e2f06a45d0beaae3377efa1dc2, State: Initialized, Role: FOLLOWER
I20260923 23:05:43.371768 12787 ts_tablet_manager.cc:1428] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:05:43.371922 12485 heartbeater.cc:503] Master 127.5.74.62:38663 was elected leader, sending a full tablet report...
I20260923 23:05:43.371948 12786 raft_consensus.cc:348] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:43.372098 12786 raft_consensus.cc:374] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:43.371872 12785 consensus_queue.cc:260] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:43.372215 12786 raft_consensus.cc:729] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc9bd0f474f425895102fae742d8082, State: Initialized, Role: FOLLOWER
I20260923 23:05:43.372385 12786 consensus_queue.cc:260] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:43.372452 12785 ts_tablet_manager.cc:1428] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 23:05:43.372888 12786 ts_tablet_manager.cc:1428] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 23:05:43.372997 12758 heartbeater.cc:503] Master 127.5.74.62:38663 was elected leader, sending a full tablet report...
I20260923 23:05:43.373030 12622 heartbeater.cc:503] Master 127.5.74.62:38663 was elected leader, sending a full tablet report...
I20260923 23:05:43.409821 12793 raft_consensus.cc:482] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:43.409976 12793 raft_consensus.cc:504] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:43.410490 12793 leader_election.cc:288] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743), 61144ca035074cf6b49ac17c874d9d42 (127.5.74.1:46765)
I20260923 23:05:43.413717 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.411155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:59095 (local address 127.5.74.1:46765)
0923 23:05:43.411206 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.411208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.411216 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:43.411246 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.411247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.411291 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.411359 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:43.411511 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.411971 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.412972 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.413354 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.413439 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.413450 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.413605 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.413640 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:05:43.413751 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.411010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:46765 (local address 127.5.74.2:59095)
0923 23:05:43.411072 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.411081 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.411109 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.411364 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.411365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.411366 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:43.411433 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.411435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.412052 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.412054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.412876 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.412879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.413436 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.413437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.413520 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.413523 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.413597 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.413640 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 23:05:43.413777 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.410961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:60881 (local address 127.5.74.3:35743)
0923 23:05:43.411036 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.411039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.411047 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:43.411079 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.411081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.411094 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.411156 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:43.411424 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.411921 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.412990 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.413357 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.413477 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.413489 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.413653 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.413690 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 23:05:43.413954 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.410739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:35743 (local address 127.5.74.2:60881)
0923 23:05:43.410878 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.410892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.410960 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.411189 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.411191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.411194 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:43.411325 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.411327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.412020 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.412021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.412878 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.412880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.413477 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.413478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.413561 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.413563 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.413646 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.413689 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":3}
I20260923 23:05:43.414292 12438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902a7d4e4ae94de49fa2197a62fcfd0c" candidate_uuid: "1bc9bd0f474f425895102fae742d8082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61144ca035074cf6b49ac17c874d9d42" is_pre_election: true
I20260923 23:05:43.414306 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902a7d4e4ae94de49fa2197a62fcfd0c" candidate_uuid: "1bc9bd0f474f425895102fae742d8082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" is_pre_election: true
I20260923 23:05:43.414450 12438 raft_consensus.cc:2454] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc9bd0f474f425895102fae742d8082 in term 0.
I20260923 23:05:43.414480 12711 raft_consensus.cc:2454] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc9bd0f474f425895102fae742d8082 in term 0.
I20260923 23:05:43.414784 12510 leader_election.cc:302] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc9bd0f474f425895102fae742d8082, cd81e0e2f06a45d0beaae3377efa1dc2; no voters:
I20260923 23:05:43.414943 12793 raft_consensus.cc:2781] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:43.415035 12793 raft_consensus.cc:482] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:43.415076 12793 raft_consensus.cc:3037] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:43.415942 12793 raft_consensus.cc:504] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:43.416121 12793 leader_election.cc:288] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [CANDIDATE]: Term 1 election: Requested vote from peers cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743), 61144ca035074cf6b49ac17c874d9d42 (127.5.74.1:46765)
I20260923 23:05:43.416411 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902a7d4e4ae94de49fa2197a62fcfd0c" candidate_uuid: "1bc9bd0f474f425895102fae742d8082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2"
I20260923 23:05:43.416491 12711 raft_consensus.cc:3037] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:43.417673 12711 raft_consensus.cc:2454] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc9bd0f474f425895102fae742d8082 in term 1.
I20260923 23:05:43.417774 12438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902a7d4e4ae94de49fa2197a62fcfd0c" candidate_uuid: "1bc9bd0f474f425895102fae742d8082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61144ca035074cf6b49ac17c874d9d42"
I20260923 23:05:43.417922 12510 leader_election.cc:302] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc9bd0f474f425895102fae742d8082, cd81e0e2f06a45d0beaae3377efa1dc2; no voters:
I20260923 23:05:43.417924 12438 raft_consensus.cc:3037] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:43.418016 12793 raft_consensus.cc:2781] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:43.418206 12793 raft_consensus.cc:686] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc9bd0f474f425895102fae742d8082, State: Running, Role: LEADER
I20260923 23:05:43.418320 12793 consensus_queue.cc:237] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
W20260923 23:05:43.418612 12624 tablet.cc:2369] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:43.419042 12438 raft_consensus.cc:2454] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc9bd0f474f425895102fae742d8082 in term 1.
I20260923 23:05:43.419445 11513 catalog_manager.cc:5495] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bc9bd0f474f425895102fae742d8082 (127.5.74.2). New cstate: current_term: 1 leader_uuid: "1bc9bd0f474f425895102fae742d8082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:43.429673 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.425524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40011 (local address 127.0.0.1:51002)
0923 23:05:43.425610 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:43.425625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:43.425687 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:43.425980 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:43.425981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:43.425982 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:43.426033 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.426035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.427202 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.427309 (+ 107us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.428526 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:43.428529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.429441 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.429442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:43.429493 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.429498 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:43.429558 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 23:05:43.429603 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 23:05:43.429718 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.425692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.5.74.60:40011)
0923 23:05:43.425771 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.425773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.425784 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:43.425816 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.425817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.425851 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.425924 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:43.426215 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.427050 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.428682 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.429322 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.429432 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.429496 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.429614 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.429658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 23:05:43.430454 12315 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.11] Dedup: 1.8->[1.9-1.11]
I20260923 23:05:43.430964 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb622f88b87c4f2c8c62912ca7a019e7. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:43.431079 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:05:43.435981 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.433179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.5.74.2:43089)
0923 23:05:43.433366 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.433369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.433381 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:05:43.433431 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.433432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.433454 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.433580 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:43.433876 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.434355 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.435144 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.435357 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.435444 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.435613 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:43.435740 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:43.435840 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.435859 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.435880 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260923 23:05:43.438663 12711 raft_consensus.cc:1261] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1bc9bd0f474f425895102fae742d8082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:43.439016 12438 raft_consensus.cc:1261] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [term 1 FOLLOWER]: Refusing update from remote peer 1bc9bd0f474f425895102fae742d8082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:43.439121 12794 consensus_queue.cc:1035] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:43.439332 12794 consensus_queue.cc:1035] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:43.442437 12801 mvcc.cc:205] Tried to move back new op lower bound from 7332678629121171456 to 7332678629042188288. Current Snapshot: MvccSnapshot[applied={T|T < 7332678629121171456}]
I20260923 23:05:43.442562 12802 mvcc.cc:205] Tried to move back new op lower bound from 7332678629121171456 to 7332678629042188288. Current Snapshot: MvccSnapshot[applied={T|T < 7332678629121171456}]
I20260923 23:05:43.442776 12803 mvcc.cc:205] Tried to move back new op lower bound from 7332678629121171456 to 7332678629042188288. Current Snapshot: MvccSnapshot[applied={T|T < 7332678629121171456}]
I20260923 23:05:43.445385 11514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:43.445477 11514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:43.446101 12259 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:43.446208 12259 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:43.446885 11514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb622f88b87c4f2c8c62912ca7a019e7: Sending DeleteTablet for 3 replicas of tablet 902a7d4e4ae94de49fa2197a62fcfd0c
I20260923 23:05:43.447245 12555 tablet_service.cc:1514] Processing DeleteTablet for tablet 902a7d4e4ae94de49fa2197a62fcfd0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:43 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:47158
I20260923 23:05:43.447661 12691 tablet_service.cc:1514] Processing DeleteTablet for tablet 902a7d4e4ae94de49fa2197a62fcfd0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:43 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:49970
I20260923 23:05:43.447809 12418 tablet_service.cc:1514] Processing DeleteTablet for tablet 902a7d4e4ae94de49fa2197a62fcfd0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:43 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260923 23:05:43.448696 12812 tablet_replica.cc:330] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: stopping tablet replica
I20260923 23:05:43.448813 12812 raft_consensus.cc:2229] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:43.448921 12812 raft_consensus.cc:2258] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:43.449311 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 11482
I20260923 23:05:43.449429 12812 ts_tablet_manager.cc:1905] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:43.452621 12812 ts_tablet_manager.cc:1918] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:43.452713 12812 log.cc:1196] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/902a7d4e4ae94de49fa2197a62fcfd0c
I20260923 23:05:43.452975 12812 ts_tablet_manager.cc:1939] T 902a7d4e4ae94de49fa2197a62fcfd0c P 1bc9bd0f474f425895102fae742d8082: Deleting consensus metadata
W20260923 23:05:43.456315 12508 connection.cc:535] server connection from 127.0.0.1:47158 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 23:05:43.457436 12814 tablet_replica.cc:330] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: stopping tablet replica
I20260923 23:05:43.457579 12814 raft_consensus.cc:2229] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:43.457731 12814 raft_consensus.cc:2258] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:43.458035 12813 tablet_replica.cc:330] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: stopping tablet replica
I20260923 23:05:43.458170 12813 raft_consensus.cc:2229] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:43.458303 12813 raft_consensus.cc:2258] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:43.458707 12814 ts_tablet_manager.cc:1905] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:43.459597 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.449070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.5.74.61:38741)
0923 23:05:43.449107 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.449109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.449116 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:05:43.449138 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.449139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.449152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.449214 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.451178 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.451954 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.453485 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.453801 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.453846 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.455692 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.455694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.455694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.455702 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.457787 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.458626 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.458628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.458628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.458687 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.458957 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.458958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.458958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.459093 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.459167 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.459409 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.459502 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:05:43.461367 12814 ts_tablet_manager.cc:1918] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:43.461458 12814 log.cc:1196] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/902a7d4e4ae94de49fa2197a62fcfd0c
I20260923 23:05:43.461726 12814 ts_tablet_manager.cc:1939] T 902a7d4e4ae94de49fa2197a62fcfd0c P 61144ca035074cf6b49ac17c874d9d42: Deleting consensus metadata
I20260923 23:05:43.461715 12813 ts_tablet_manager.cc:1905] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 23:05:43.462363 12372 connection.cc:441] server connection from 127.0.0.1:44118 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44118 (request call id 1) could send its response
I20260923 23:05:43.463311 12813 ts_tablet_manager.cc:1918] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:43.463466 12813 log.cc:1196] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/902a7d4e4ae94de49fa2197a62fcfd0c
I20260923 23:05:43.463819 12813 ts_tablet_manager.cc:1939] T 902a7d4e4ae94de49fa2197a62fcfd0c P cd81e0e2f06a45d0beaae3377efa1dc2: Deleting consensus metadata
W20260923 23:05:43.464282 12644 connection.cc:441] server connection from 127.0.0.1:49970 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49970 (request call id 1) could send its response
I20260923 23:05:43.468937 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.448995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.5.74.60:40011)
0923 23:05:43.449039 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.449041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.460135 (+ 11094us) server_negotiation.cc:407] Connection header received
0923 23:05:43.460264 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.460265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.460275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.460306 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.460988 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.461729 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.462922 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.463229 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.463295 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.464992 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.464993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.464994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.465002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.466947 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.467503 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.467505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.467506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.467564 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.467820 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.467821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.467821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.467983 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.468071 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.468662 (+ 591us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.468799 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:43.471815 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.457665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.5.74.61:38741)
0923 23:05:43.457712 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.457714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.458385 (+ 671us) server_negotiation.cc:407] Connection header received
0923 23:05:43.458738 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.458739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.458753 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.458789 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.460465 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.461275 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.466243 (+ 4968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.466661 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.466726 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.468671 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.468672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.468673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.468680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.470484 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.470991 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.470992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.470993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.471054 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.471263 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.471264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.471264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.471394 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.471514 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.471612 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.471693 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:05:43.472014 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:43.458086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.5.74.60:40011)
0923 23:05:43.458129 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:43.458131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:43.458684 (+ 553us) server_negotiation.cc:407] Connection header received
0923 23:05:43.459602 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:43.459603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:43.459616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:43.459650 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:43.464145 (+ 4495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.465021 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.466155 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:43.466466 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:43.466537 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:43.467987 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:43.467989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:43.467989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:43.467996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:43.470763 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.471183 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.471184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.471185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.471238 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:43.471455 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:43.471456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:43.471456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:43.471585 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:43.471653 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:43.471848 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 23:05:43.471924 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:44.444604 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.444349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.1:58157)
0923 23:05:44.444464 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.444551 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260923 23:05:44.444726 12824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.444512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:59177)
0923 23:05:44.444605 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.444670 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":49,"threads_started":1}
W20260923 23:05:44.444860 12485 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:38663 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
W20260923 23:05:44.444994 12622 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:38663 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
I20260923 23:05:44.445943 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.445682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.3:59673)
0923 23:05:44.445792 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.445895 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
W20260923 23:05:44.446136 12758 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:38663 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
I20260923 23:05:44.947888 12826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fb622f88b87c4f2c8c62912ca7a019e7)
I20260923 23:05:44.948011 12827 raft_consensus.cc:482] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fb622f88b87c4f2c8c62912ca7a019e7)
I20260923 23:05:44.948038 12826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:44.948117 12827 raft_consensus.cc:504] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:44.948376 12826 leader_election.cc:288] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663), d123501b5751435baa822d815d2d5b86 (127.5.74.60:40011)
I20260923 23:05:44.948647 12827 leader_election.cc:288] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663), 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61:38741)
I20260923 23:05:44.948846 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.948532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60154)
0923 23:05:44.948631 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.948787 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260923 23:05:44.948860 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.948532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60156)
0923 23:05:44.948769 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.948816 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":49,"threads_started":1}
W20260923 23:05:44.948984 12099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
W20260923 23:05:44.949134 12291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
I20260923 23:05:44.949775 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.949655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60182)
0923 23:05:44.949713 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.949746 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 23:05:44.949847 12291 leader_election.cc:334] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663): Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
I20260923 23:05:44.950110 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.949983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60198)
0923 23:05:44.950045 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.950080 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 23:05:44.950237 12099 leader_election.cc:334] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663): Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
I20260923 23:05:44.952517 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.948782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.5.74.60:40011)
0923 23:05:44.949102 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:44.949104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:44.949112 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:44.949139 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:44.949140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:44.949153 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:44.949240 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:44.949598 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:44.950570 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:44.951536 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:44.952019 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:44.952094 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:44.952105 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:44.952389 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 23:05:44.952430 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":49,"threads_started":1}
I20260923 23:05:44.952524 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.948863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40011 (local address 127.0.0.1:51022)
0923 23:05:44.948923 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.948934 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:44.948993 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:44.949248 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:44.949248 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:44.949251 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:44.949504 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:44.949508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:44.950670 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:44.950671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:44.951430 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:44.951432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:44.952121 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:44.952122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:44.952239 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:44.952309 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 23:05:44.952374 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 23:05:44.952412 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"thread_start_us":47,"threads_started":1}
I20260923 23:05:44.952780 12315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3efbc7de087342ba8b39f238f8c23a37" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d123501b5751435baa822d815d2d5b86" is_pre_election: true
I20260923 23:05:44.952888 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.948798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:38741 (local address 127.0.0.1:58942)
0923 23:05:44.949092 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.949102 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:44.949204 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:44.949613 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:44.949614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:44.949616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:44.949660 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:44.949662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:44.950684 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:44.950685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:44.951733 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:44.951735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:44.952558 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:44.952560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:44.952728 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:44.952733 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:44.952784 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:05:44.952828 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":46,"threads_started":1}
I20260923 23:05:44.952895 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.948932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.5.74.61:38741)
0923 23:05:44.949023 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:44.949025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:44.949157 (+ 132us) server_negotiation.cc:407] Connection header received
0923 23:05:44.949288 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:44.949417 (+ 129us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:44.949437 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:44.949537 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:44.949744 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:44.950591 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:44.951924 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:44.952459 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:44.952534 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:44.952546 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:44.952790 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 23:05:44.952835 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":49,"threads_started":1}
I20260923 23:05:44.952908 12315 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efbc7de087342ba8b39f238f8c23a37 in term 1.
I20260923 23:05:44.953068 12099 leader_election.cc:302] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3efbc7de087342ba8b39f238f8c23a37, d123501b5751435baa822d815d2d5b86; no voters: fb622f88b87c4f2c8c62912ca7a019e7
I20260923 23:05:44.953150 12123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d123501b5751435baa822d815d2d5b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3efbc7de087342ba8b39f238f8c23a37" is_pre_election: true
I20260923 23:05:44.953284 12826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 23:05:44.953400 12123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d123501b5751435baa822d815d2d5b86 in term 1.
I20260923 23:05:44.953459 12826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fb622f88b87c4f2c8c62912ca7a019e7)
I20260923 23:05:44.953503 12826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:05:44.953603 12290 leader_election.cc:302] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3efbc7de087342ba8b39f238f8c23a37, d123501b5751435baa822d815d2d5b86; no voters: fb622f88b87c4f2c8c62912ca7a019e7
I20260923 23:05:44.953712 12827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 23:05:44.953751 12827 raft_consensus.cc:482] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fb622f88b87c4f2c8c62912ca7a019e7)
I20260923 23:05:44.953770 12827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:05:44.954411 12826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:44.954571 12826 leader_election.cc:288] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [CANDIDATE]: Term 2 election: Requested vote from peers fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663), d123501b5751435baa822d815d2d5b86 (127.5.74.60:40011)
I20260923 23:05:44.954655 12827 raft_consensus.cc:504] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:44.954807 12827 leader_election.cc:288] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 2 election: Requested vote from peers fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663), 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61:38741)
I20260923 23:05:44.954820 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.954709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60214)
0923 23:05:44.954762 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.954792 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 23:05:44.954934 12315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3efbc7de087342ba8b39f238f8c23a37" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d123501b5751435baa822d815d2d5b86"
I20260923 23:05:44.955010 12315 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3efbc7de087342ba8b39f238f8c23a37 in current term 2: Already voted for candidate d123501b5751435baa822d815d2d5b86 in this term.
I20260923 23:05:44.955052 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.954879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60222)
0923 23:05:44.954980 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.955008 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 23:05:44.955168 12123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d123501b5751435baa822d815d2d5b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3efbc7de087342ba8b39f238f8c23a37"
I20260923 23:05:44.955278 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.955017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60228)
0923 23:05:44.955207 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.955236 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260923 23:05:44.955241 12123 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d123501b5751435baa822d815d2d5b86 in current term 2: Already voted for candidate 3efbc7de087342ba8b39f238f8c23a37 in this term.
W20260923 23:05:44.955336 12099 leader_election.cc:334] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663): Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
I20260923 23:05:44.955390 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:44.955318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60236)
0923 23:05:44.955352 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:44.955371 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260923 23:05:44.955407 12099 leader_election.cc:302] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efbc7de087342ba8b39f238f8c23a37; no voters: d123501b5751435baa822d815d2d5b86, fb622f88b87c4f2c8c62912ca7a019e7
W20260923 23:05:44.955466 12291 leader_election.cc:334] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663): Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
I20260923 23:05:44.955507 12290 leader_election.cc:302] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d123501b5751435baa822d815d2d5b86; no voters: 3efbc7de087342ba8b39f238f8c23a37, fb622f88b87c4f2c8c62912ca7a019e7
I20260923 23:05:44.955528 12826 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 23:05:44.955592 12827 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 23:05:45.445464 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:45.445216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.1:34463)
0923 23:05:45.445361 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:45.445408 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260923 23:05:45.445488 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:45.445260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:37813)
0923 23:05:45.445386 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:45.445442 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":89,"threads_started":1}
I20260923 23:05:45.446679 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:45.446398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.3:45385)
0923 23:05:45.446569 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:45.446623 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
I20260923 23:05:46.446128 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.445913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:52219)
0923 23:05:46.446037 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.446078 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260923 23:05:46.446131 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.445913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.1:44995)
0923 23:05:46.446037 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.446078 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260923 23:05:46.447278 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.447095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.3:34017)
0923 23:05:46.447204 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.447240 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260923 23:05:46.580511 12842 raft_consensus.cc:482] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:46.580641 12842 raft_consensus.cc:504] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:46.581012 12842 leader_election.cc:288] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663), 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61:38741)
I20260923 23:05:46.581238 12123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d123501b5751435baa822d815d2d5b86" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3efbc7de087342ba8b39f238f8c23a37" is_pre_election: true
I20260923 23:05:46.581324 12123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d123501b5751435baa822d815d2d5b86 in term 2.
I20260923 23:05:46.581333 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.581124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60498)
0923 23:05:46.581239 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.581278 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":77,"threads_started":1}
I20260923 23:05:46.581475 12290 leader_election.cc:302] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efbc7de087342ba8b39f238f8c23a37, d123501b5751435baa822d815d2d5b86; no voters:
I20260923 23:05:46.581568 12842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 23:05:46.581610 12842 raft_consensus.cc:482] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:46.581635 12842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 2 FOLLOWER]: Advancing to term 3
I20260923 23:05:46.581772 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.581681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60504)
0923 23:05:46.581715 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.581745 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260923 23:05:46.581883 12291 leader_election.cc:334] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663): Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
I20260923 23:05:46.582430 12842 raft_consensus.cc:504] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:46.582576 12842 leader_election.cc:288] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 3 election: Requested vote from peers fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663), 3efbc7de087342ba8b39f238f8c23a37 (127.5.74.61:38741)
I20260923 23:05:46.582765 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.582672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60518)
0923 23:05:46.582708 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.582734 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260923 23:05:46.582758 12123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d123501b5751435baa822d815d2d5b86" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3efbc7de087342ba8b39f238f8c23a37"
I20260923 23:05:46.582841 12123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 2 FOLLOWER]: Advancing to term 3
I20260923 23:05:46.583142 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.583048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60522)
0923 23:05:46.583097 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.583122 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:05:46.583254 12291 leader_election.cc:334] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663): Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
I20260923 23:05:46.583575 12123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d123501b5751435baa822d815d2d5b86 in term 3.
I20260923 23:05:46.583735 12290 leader_election.cc:302] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3efbc7de087342ba8b39f238f8c23a37, d123501b5751435baa822d815d2d5b86; no voters: fb622f88b87c4f2c8c62912ca7a019e7
I20260923 23:05:46.583832 12842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 23:05:46.583940 12842 raft_consensus.cc:686] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [term 3 LEADER]: Becoming Leader. State: Replica: d123501b5751435baa822d815d2d5b86, State: Running, Role: LEADER
I20260923 23:05:46.584019 12842 consensus_queue.cc:237] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } }
I20260923 23:05:46.584403 12845 sys_catalog.cc:455] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d123501b5751435baa822d815d2d5b86. Latest consensus state: current_term: 3 leader_uuid: "d123501b5751435baa822d815d2d5b86" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:46.584512 12845 sys_catalog.cc:458] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:46.584693 12847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:46.585268 12847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:46.585350 12847 catalog_manager.cc:1259] Loaded cluster ID: 392d22574c744aaa838e1b7ef1304bff
I20260923 23:05:46.585384 12847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:46.585580 12847 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:46.585665 12847 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Loaded TSK: 0
I20260923 23:05:46.585808 12847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:05:46.646765 12305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:46.648581 12305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:46.649717 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.649583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60536)
0923 23:05:46.649642 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.649682 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 23:05:46.649701 12123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [term 3 FOLLOWER]: Refusing update from remote peer d123501b5751435baa822d815d2d5b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 23:05:46.649971 12842 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 23:05:46.650182 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.650013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.0.0.1:60540)
0923 23:05:46.650115 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.650150 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260923 23:05:46.650267 12291 consensus_peers.cc:487] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 -> Peer fb622f88b87c4f2c8c62912ca7a019e7 (127.5.74.62:38663): Couldn't send request to peer fb622f88b87c4f2c8c62912ca7a019e7. Status: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 23:05:46.650549 12849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d123501b5751435baa822d815d2d5b86. Latest consensus state: current_term: 3 leader_uuid: "d123501b5751435baa822d815d2d5b86" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:46.650651 12849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:05:46.651116 12849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d123501b5751435baa822d815d2d5b86" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:46.651276 12849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efbc7de087342ba8b39f238f8c23a37 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:05:46.651295 12845 sys_catalog.cc:455] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d123501b5751435baa822d815d2d5b86" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb622f88b87c4f2c8c62912ca7a019e7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38663 } } peers { permanent_uuid: "3efbc7de087342ba8b39f238f8c23a37" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 38741 } attrs { promote: false } } peers { permanent_uuid: "d123501b5751435baa822d815d2d5b86" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40011 } attrs { promote: false } } }
I20260923 23:05:46.651392 12845 sys_catalog.cc:458] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:46.656628 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.653436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.5.74.2:43089)
0923 23:05:46.653626 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:46.653628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:46.653637 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:46.653663 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:46.653664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:46.653677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:46.653906 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:46.653949 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.654534 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.655692 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.656299 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.656367 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:46.656381 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:46.656524 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 23:05:46.656564 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":138,"threads_started":1}
I20260923 23:05:46.656626 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.653241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:43089 (local address 127.0.0.1:47178)
0923 23:05:46.653280 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.653290 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:46.653336 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:46.653773 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:46.653774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:46.653775 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:46.653836 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:46.653837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.654618 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.654619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:46.655599 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:46.655601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.656377 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.656377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:46.656443 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:46.656446 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:46.656507 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:05:46.656551 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 23:05:46.657140 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.653250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:35743 (local address 127.0.0.1:49976)
0923 23:05:46.653427 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.653438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:46.653492 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:46.653983 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:46.653984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:46.653986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:46.654032 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:46.654034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.655043 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.655044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:46.656006 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:46.656008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.656856 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.656858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:46.656921 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:46.656926 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:46.656957 (+ 31us) client_negotiation.cc:240] Negotiation successful
0923 23:05:46.657032 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":49,"thread_start_us":134,"threads_started":1}
I20260923 23:05:46.657158 12555 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9e9ca11f564dac83e414307d52d7f4 (DEFAULT_TABLE table=test-table [id=cc4887d96ceb4caeb23a1b9d2cd56570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:46.657271 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.653516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:46765 (local address 127.0.0.1:44130)
0923 23:05:46.653630 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.653640 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:46.653699 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:46.654083 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:46.654084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:46.654086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:46.654121 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:46.654122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.655039 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.655040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:46.656230 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:46.656233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.656973 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.656974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:46.657040 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:46.657044 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:46.657108 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:05:46.657167 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":20,"thread_start_us":69,"threads_started":1}
I20260923 23:05:46.657326 12555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9e9ca11f564dac83e414307d52d7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:46.657406 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.653331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.5.74.3:35743)
0923 23:05:46.653834 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:46.653836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:46.653844 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:46.653872 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:46.653873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:46.653883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:46.654096 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:46.654139 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.654937 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.656070 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.656675 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.657253 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:46.657267 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:46.657288 (+ 21us) server_negotiation.cc:299] Negotiation successful
0923 23:05:46.657329 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":53,"threads_started":1}
I20260923 23:05:46.657480 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.653680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.5.74.1:46765)
0923 23:05:46.653933 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:46.653936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:46.653944 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:46.653967 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:46.653968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:46.653978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:46.654041 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:46.654204 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.654959 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.656313 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.656883 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.656970 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:46.656981 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:46.657371 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 23:05:46.657415 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":54,"threads_started":1}
I20260923 23:05:46.657575 12691 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9e9ca11f564dac83e414307d52d7f4 (DEFAULT_TABLE table=test-table [id=cc4887d96ceb4caeb23a1b9d2cd56570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:46.657665 12691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9e9ca11f564dac83e414307d52d7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:46.657779 12418 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9e9ca11f564dac83e414307d52d7f4 (DEFAULT_TABLE table=test-table [id=cc4887d96ceb4caeb23a1b9d2cd56570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:46.657893 12418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9e9ca11f564dac83e414307d52d7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:46.658871 12860 tablet_bootstrap.cc:492] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: Bootstrap starting.
I20260923 23:05:46.659109 12861 tablet_bootstrap.cc:492] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: Bootstrap starting.
I20260923 23:05:46.659274 12862 tablet_bootstrap.cc:492] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42: Bootstrap starting.
I20260923 23:05:46.659428 12860 tablet_bootstrap.cc:654] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:46.659509 12861 tablet_bootstrap.cc:654] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:46.659672 12862 tablet_bootstrap.cc:654] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:46.660022 12860 tablet_bootstrap.cc:492] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: No bootstrap required, opened a new log
I20260923 23:05:46.660081 12860 ts_tablet_manager.cc:1397] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:46.660192 12861 tablet_bootstrap.cc:492] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: No bootstrap required, opened a new log
I20260923 23:05:46.660239 12861 ts_tablet_manager.cc:1397] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:46.660233 12860 raft_consensus.cc:348] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:46.660303 12862 tablet_bootstrap.cc:492] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42: No bootstrap required, opened a new log
I20260923 23:05:46.660331 12860 raft_consensus.cc:374] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:46.660348 12862 ts_tablet_manager.cc:1397] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:46.660364 12860 raft_consensus.cc:729] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc9bd0f474f425895102fae742d8082, State: Initialized, Role: FOLLOWER
I20260923 23:05:46.660409 12860 consensus_queue.cc:260] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:46.660403 12861 raft_consensus.cc:348] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:46.660463 12861 raft_consensus.cc:374] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:46.660490 12861 raft_consensus.cc:729] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd81e0e2f06a45d0beaae3377efa1dc2, State: Initialized, Role: FOLLOWER
I20260923 23:05:46.660458 12862 raft_consensus.cc:348] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:46.660518 12862 raft_consensus.cc:374] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:46.660538 12862 raft_consensus.cc:729] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61144ca035074cf6b49ac17c874d9d42, State: Initialized, Role: FOLLOWER
I20260923 23:05:46.660555 12861 consensus_queue.cc:260] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:46.660578 12860 ts_tablet_manager.cc:1428] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:46.660604 12862 consensus_queue.cc:260] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:46.660727 12861 ts_tablet_manager.cc:1428] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:46.660983 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.660844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:40265)
0923 23:05:46.660921 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.660957 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 23:05:46.661363 12862 ts_tablet_manager.cc:1428] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:46.661544 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.661431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.3:32779)
0923 23:05:46.661477 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.661511 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 23:05:46.661592 12757 heartbeater.cc:503] Master 127.5.74.60:40011 was elected leader, sending a full tablet report...
I20260923 23:05:46.661688 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.661585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:48003)
0923 23:05:46.661623 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.661664 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 23:05:46.661895 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.661809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.3:46813)
0923 23:05:46.661843 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.661867 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260923 23:05:46.662312 12484 heartbeater.cc:503] Master 127.5.74.60:40011 was elected leader, sending a full tablet report...
I20260923 23:05:46.662366 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.662175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.1:34363)
0923 23:05:46.662305 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.662333 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 23:05:46.662694 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.662598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.1:49397)
0923 23:05:46.662638 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.662665 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 23:05:46.675994 12866 raft_consensus.cc:482] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:46.676087 12866 raft_consensus.cc:504] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:46.676290 12866 leader_election.cc:288] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743), 61144ca035074cf6b49ac17c874d9d42 (127.5.74.1:46765)
I20260923 23:05:46.676508 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9e9ca11f564dac83e414307d52d7f4" candidate_uuid: "1bc9bd0f474f425895102fae742d8082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" is_pre_election: true
I20260923 23:05:46.676587 12711 raft_consensus.cc:2454] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc9bd0f474f425895102fae742d8082 in term 0.
I20260923 23:05:46.676525 12438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9e9ca11f564dac83e414307d52d7f4" candidate_uuid: "1bc9bd0f474f425895102fae742d8082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61144ca035074cf6b49ac17c874d9d42" is_pre_election: true
I20260923 23:05:46.676646 12438 raft_consensus.cc:2454] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bc9bd0f474f425895102fae742d8082 in term 0.
I20260923 23:05:46.676832 12510 leader_election.cc:302] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc9bd0f474f425895102fae742d8082, 61144ca035074cf6b49ac17c874d9d42; no voters:
I20260923 23:05:46.676942 12866 raft_consensus.cc:2781] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:46.676991 12866 raft_consensus.cc:482] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:46.677022 12866 raft_consensus.cc:3037] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:46.677770 12866 raft_consensus.cc:504] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:46.677914 12866 leader_election.cc:288] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [CANDIDATE]: Term 1 election: Requested vote from peers cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743), 61144ca035074cf6b49ac17c874d9d42 (127.5.74.1:46765)
I20260923 23:05:46.678074 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9e9ca11f564dac83e414307d52d7f4" candidate_uuid: "1bc9bd0f474f425895102fae742d8082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2"
I20260923 23:05:46.678100 12438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9e9ca11f564dac83e414307d52d7f4" candidate_uuid: "1bc9bd0f474f425895102fae742d8082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61144ca035074cf6b49ac17c874d9d42"
I20260923 23:05:46.678164 12438 raft_consensus.cc:3037] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:46.678177 12711 raft_consensus.cc:3037] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:46.678748 12438 raft_consensus.cc:2454] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc9bd0f474f425895102fae742d8082 in term 1.
I20260923 23:05:46.678907 12711 raft_consensus.cc:2454] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc9bd0f474f425895102fae742d8082 in term 1.
I20260923 23:05:46.678937 12510 leader_election.cc:302] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc9bd0f474f425895102fae742d8082, 61144ca035074cf6b49ac17c874d9d42; no voters:
I20260923 23:05:46.679036 12866 raft_consensus.cc:2781] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:46.679200 12866 raft_consensus.cc:686] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc9bd0f474f425895102fae742d8082, State: Running, Role: LEADER
I20260923 23:05:46.679316 12621 heartbeater.cc:503] Master 127.5.74.60:40011 was elected leader, sending a full tablet report...
I20260923 23:05:46.679247 12866 consensus_queue.cc:237] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } }
I20260923 23:05:46.679584 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.679448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:42285)
0923 23:05:46.679520 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.679557 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 23:05:46.680210 12303 catalog_manager.cc:5495] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bc9bd0f474f425895102fae742d8082 (127.5.74.2). New cstate: current_term: 1 leader_uuid: "1bc9bd0f474f425895102fae742d8082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc9bd0f474f425895102fae742d8082" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:46.694777 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.691793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.5.74.2:43089)
0923 23:05:46.691861 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:46.691863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:46.691873 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:46.691931 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:46.691932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:46.691946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:46.692026 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:46.692369 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.693067 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.693945 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:46.694196 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:46.694304 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:46.694497 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:46.694531 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:46.694602 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:46.694664 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 23:05:46.694686 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:05:46.695823 12711 raft_consensus.cc:1261] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1bc9bd0f474f425895102fae742d8082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:46.695824 12438 raft_consensus.cc:1261] T 2f9e9ca11f564dac83e414307d52d7f4 P 61144ca035074cf6b49ac17c874d9d42 [term 1 FOLLOWER]: Refusing update from remote peer 1bc9bd0f474f425895102fae742d8082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:46.696120 12869 consensus_queue.cc:1035] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd81e0e2f06a45d0beaae3377efa1dc2" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:46.696249 12866 consensus_queue.cc:1035] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61144ca035074cf6b49ac17c874d9d42" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:46.697599 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.697459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.3:34989)
0923 23:05:46.697535 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.697564 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 23:05:46.697599 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.697363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.1:56515)
0923 23:05:46.697526 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.697562 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 23:05:46.698499 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.698369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:48923)
0923 23:05:46.698435 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.698467 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 23:05:46.698964 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.698385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.1:57983)
0923 23:05:46.698906 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.698936 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504}
I20260923 23:05:46.699723 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.699594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.2:38009)
0923 23:05:46.699668 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.699698 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 23:05:46.700114 12303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:46.700235 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:46.700090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:38663 (local address 127.5.74.3:54297)
0923 23:05:46.700179 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.700204 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 23:05:46.700241 12303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:46.701632 12303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d123501b5751435baa822d815d2d5b86: Sending DeleteTablet for 3 replicas of tablet 2f9e9ca11f564dac83e414307d52d7f4
I20260923 23:05:46.701817 12691 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9e9ca11f564dac83e414307d52d7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:46 UTC) from {username='oryx', principal='oryx/127.5.74.60@KRBTEST.COM'} at 127.0.0.1:49976
I20260923 23:05:46.701974 12555 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9e9ca11f564dac83e414307d52d7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:46 UTC) from {username='oryx', principal='oryx/127.5.74.60@KRBTEST.COM'} at 127.0.0.1:47178
I20260923 23:05:46.702025 12880 tablet_replica.cc:330] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: stopping tablet replica
I20260923 23:05:46.702137 12881 tablet_replica.cc:330] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: stopping tablet replica
I20260923 23:05:46.702116 12418 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9e9ca11f564dac83e414307d52d7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:46 UTC) from {username='oryx', principal='oryx/127.5.74.60@KRBTEST.COM'} at 127.0.0.1:44130
I20260923 23:05:46.702153 12880 raft_consensus.cc:2229] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:46.702180 12881 raft_consensus.cc:2229] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:46.702201 12880 raft_consensus.cc:2258] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:46.702239 12881 raft_consensus.cc:2258] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:46.702065 12691 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9e9ca11f564dac83e414307d52d7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:46 UTC) from {username='oryx', principal='oryx/127.5.74.60@KRBTEST.COM'} at 127.0.0.1:49976
I20260923 23:05:46.702420 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 12266
I20260923 23:05:46.704384 12881 ts_tablet_manager.cc:1905] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:46.705344 12880 ts_tablet_manager.cc:1905] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:46.706266 12881 ts_tablet_manager.cc:1918] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:46.706365 12881 log.cc:1196] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/2f9e9ca11f564dac83e414307d52d7f4
I20260923 23:05:46.706665 12881 ts_tablet_manager.cc:1939] T 2f9e9ca11f564dac83e414307d52d7f4 P 1bc9bd0f474f425895102fae742d8082: Deleting consensus metadata
I20260923 23:05:46.707055 12292 catalog_manager.cc:4841] TS 1bc9bd0f474f425895102fae742d8082 (127.5.74.2:43089): tablet 2f9e9ca11f564dac83e414307d52d7f4 (table test-table [id=cc4887d96ceb4caeb23a1b9d2cd56570]) successfully deleted
I20260923 23:05:46.707135 12880 ts_tablet_manager.cc:1918] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:46.707233 12880 log.cc:1196] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/2f9e9ca11f564dac83e414307d52d7f4
I20260923 23:05:46.707463 12880 ts_tablet_manager.cc:1939] T 2f9e9ca11f564dac83e414307d52d7f4 P cd81e0e2f06a45d0beaae3377efa1dc2: Deleting consensus metadata
I20260923 23:05:46.708139 12291 catalog_manager.cc:4841] TS cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743): tablet 2f9e9ca11f564dac83e414307d52d7f4 (table test-table [id=cc4887d96ceb4caeb23a1b9d2cd56570]) successfully deleted
W20260923 23:05:46.708446 12858 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:46.708068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:46765 (local address 127.0.0.1:44134)
0923 23:05:46.708349 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:46.708379 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:46765: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":264}
I20260923 23:05:46.708467 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 12492
W20260923 23:05:46.708580 12291 catalog_manager.cc:4584] TS 61144ca035074cf6b49ac17c874d9d42 (127.5.74.1:46765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f9e9ca11f564dac83e414307d52d7f4: Network error: Client connection negotiation failed: client connection to 127.5.74.1:46765: connect: Connection reset by peer (error 104)
W20260923 23:05:46.710604 12291 catalog_manager.cc:4808] TS cd81e0e2f06a45d0beaae3377efa1dc2 (127.5.74.3:35743): delete failed for tablet 2f9e9ca11f564dac83e414307d52d7f4 because the tablet was not found. No further retry: Not found: Tablet not found: 2f9e9ca11f564dac83e414307d52d7f4
I20260923 23:05:46.714841 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 12628
I20260923 23:05:46.719732 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 11978
I20260923 23:05:46.724979 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 12101
2026-09-23T23:05:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6406 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:46 dist-test-slave-j4c5 krb5kdc[12885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:46 dist-test-slave-j4c5 krb5kdc[12885](info): set up 2 sockets
Sep 23 23:05:46 dist-test-slave-j4c5 krb5kdc[12885](info): commencing operation
krb5kdc: starting...
W20260923 23:05:47.848909 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:47 dist-test-slave-j4c5 krb5kdc[12885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.62.keytab.
Entry for principal oryx/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.62.keytab.
I20260923 23:05:48.002166 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34937
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.62.keytab
--principal=oryx/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:34937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:48.099578 12898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:48.099880 12898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:48.099925 12898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:48.101446 12898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:48.101534 12898 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 23:05:48.101560 12898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:48.101585 12898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:48.101620 12898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:48.101652 12898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:48.103500 12898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:34937
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.62.keytab
--principal=oryx/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:48.103842 12898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:48.104108 12898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:48.110270 12904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:48.110275 12903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:48.110311 12898 server_base.cc:1034] running on GCE node
W20260923 23:05:48.110275 12906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:48.110805 12898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:48.111148 12898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:48.112346 12898 hybrid_clock.cc:648] HybridClock initialized: now 1790204748112279 us; error 91 us; skew 500 ppm
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:48.126049 12898 init.cc:377] Logged in from keytab as oryx/127.5.74.62@KRBTEST.COM (short username oryx)
I20260923 23:05:48.126645 12898 webserver.cc:460] Webserver started at http://127.5.74.62:38521/ using document root <none> and password file <none>
I20260923 23:05:48.126891 12898 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:48.126940 12898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:48.127086 12898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:48.128276 12898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "83925fa9376848f3a1bbc2b65be60879"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "f61755e620a62d0fa2687a88ae8822ce"
server_key_iv: "c20af6488b31c8fa887d9b4aeff40e28"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.128674 12898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "83925fa9376848f3a1bbc2b65be60879"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "f61755e620a62d0fa2687a88ae8822ce"
server_key_iv: "c20af6488b31c8fa887d9b4aeff40e28"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.130172 12898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:48.131145 12913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:48.131470 12898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:48.131579 12898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "83925fa9376848f3a1bbc2b65be60879"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "f61755e620a62d0fa2687a88ae8822ce"
server_key_iv: "c20af6488b31c8fa887d9b4aeff40e28"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.131700 12898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:48.140609 12898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:48.142999 12898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:48.143160 12898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:48.148481 12898 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:34937
I20260923 23:05:48.148530 12975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:34937 every 8 connection(s)
I20260923 23:05:48.148984 12898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:48.149696 12976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:48.152318 12976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879: Bootstrap starting.
I20260923 23:05:48.152913 12976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:48.153155 12976 log.cc:824] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:48.153993 12976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879: No bootstrap required, opened a new log
I20260923 23:05:48.156112 12976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83925fa9376848f3a1bbc2b65be60879" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34937 } }
I20260923 23:05:48.156442 12976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:48.156472 12976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83925fa9376848f3a1bbc2b65be60879, State: Initialized, Role: FOLLOWER
I20260923 23:05:48.156611 12976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83925fa9376848f3a1bbc2b65be60879" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34937 } }
I20260923 23:05:48.156708 12976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:48.156733 12976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:48.156788 12976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:48.157332 12976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83925fa9376848f3a1bbc2b65be60879" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34937 } }
I20260923 23:05:48.157426 12976 leader_election.cc:302] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83925fa9376848f3a1bbc2b65be60879; no voters:
I20260923 23:05:48.157603 12976 leader_election.cc:288] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:48.157635 12979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:48.157766 12979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [term 1 LEADER]: Becoming Leader. State: Replica: 83925fa9376848f3a1bbc2b65be60879, State: Running, Role: LEADER
I20260923 23:05:48.157876 12976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:48.157936 12979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83925fa9376848f3a1bbc2b65be60879" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34937 } }
I20260923 23:05:48.158347 12980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83925fa9376848f3a1bbc2b65be60879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83925fa9376848f3a1bbc2b65be60879" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34937 } } }
I20260923 23:05:48.158633 12980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:48.158530 12981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83925fa9376848f3a1bbc2b65be60879. Latest consensus state: current_term: 1 leader_uuid: "83925fa9376848f3a1bbc2b65be60879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83925fa9376848f3a1bbc2b65be60879" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34937 } } }
I20260923 23:05:48.158859 12981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:48.158962 12988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:48.159022 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 12898
I20260923 23:05:48.159145 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:48.159404 5416 external_mini_cluster.cc:1428] Setting key dc3d7fcc0a8c0725884250a284a208e4
I20260923 23:05:48.159773 12988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:48.161695 12988 catalog_manager.cc:1347] Generated new cluster ID: 1f49f7566b474a68a9791a398abde3f0
I20260923 23:05:48.161751 12988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:48.172580 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.159911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.5.74.62:34937)
0923 23:05:48.160242 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.160252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.160287 (+ 35us) server_negotiation.cc:407] Connection header received
0923 23:05:48.160562 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.160570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.160634 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.160783 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:48.161536 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.163104 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.164385 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.165031 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.165142 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.168273 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:48.168278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:48.168279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:48.168295 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:48.170549 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.170983 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.170988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.170989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.171127 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.171346 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.171347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.171347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.171717 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:48.171889 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.171973 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.172292 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":83,"threads_started":1}
I20260923 23:05:48.189086 12988 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:48.189739 12988 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:48.195989 12988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879: Generated new TSK 0
I20260923 23:05:48.196223 12988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab.
Entry for principal oryx/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab.
I20260923 23:05:48.233819 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34937
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab
--principal=oryx/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:48.338909 13002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:48.339157 13002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:48.339184 13002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:48.340615 13002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:48.340670 13002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:48.340694 13002 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:05:48.340714 13002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:48.342657 13002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab
--principal=oryx/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.74.62:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:48.342923 13002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:48.343124 13002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:48.348937 13007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:48.348937 13008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:48.348932 13010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:48.348999 13002 server_base.cc:1034] running on GCE node
I20260923 23:05:48.349474 13002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:48.349773 13002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:48.351004 13002 hybrid_clock.cc:648] HybridClock initialized: now 1790204748350995 us; error 44 us; skew 500 ppm
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:48.354621 13002 init.cc:377] Logged in from keytab as oryx/127.5.74.1@KRBTEST.COM (short username oryx)
I20260923 23:05:48.355254 13002 webserver.cc:460] Webserver started at http://127.5.74.1:36989/ using document root <none> and password file <none>
I20260923 23:05:48.355597 13002 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:48.355690 13002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:48.355859 13002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:48.357172 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "6fb06458ffec40c897035e161d13175b"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "de550fb7d06a86fad3e082a8eef499c0"
server_key_iv: "30dcdc0e8313cec03789c3dc626b72c7"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.357594 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "6fb06458ffec40c897035e161d13175b"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "de550fb7d06a86fad3e082a8eef499c0"
server_key_iv: "30dcdc0e8313cec03789c3dc626b72c7"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.359531 13002 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 23:05:48.360502 13017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:48.360702 13002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:48.360818 13002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "6fb06458ffec40c897035e161d13175b"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "de550fb7d06a86fad3e082a8eef499c0"
server_key_iv: "30dcdc0e8313cec03789c3dc626b72c7"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.360899 13002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:48.379308 13002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:48.381994 13002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:48.382189 13002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:48.382901 13002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:48.382968 13002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:48.383008 13002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:48.383023 13002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:48.390972 13002 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:39301
I20260923 23:05:48.391086 13132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:39301 every 8 connection(s)
I20260923 23:05:48.391429 13002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:48.394380 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.383379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:59405 (local address 127.5.74.62:34937)
0923 23:05:48.383438 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.383441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.383756 (+ 315us) server_negotiation.cc:407] Connection header received
0923 23:05:48.384362 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.384364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.384379 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.384468 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:48.385300 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.385822 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.386970 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.387191 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.387286 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.390451 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:48.390453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:48.390454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:48.390467 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:48.392615 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.393091 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.393093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.393093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.393171 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.393473 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.393475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.393475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.393622 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:48.393790 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.394173 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.394262 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:05:48.394644 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.383422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34937 (local address 127.5.74.1:59405)
0923 23:05:48.383610 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:48.383653 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:48.384197 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:48.384658 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:48.384663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:48.384965 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:48.385170 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.385175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.385938 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.385939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.386866 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.386869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.387304 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.387305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.387467 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.388227 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:48.388241 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:48.390325 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:48.392714 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.392720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.392721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.392998 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.393251 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.393252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.393252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.393370 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.393806 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:48.393814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:48.394001 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 23:05:48.394141 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 23:05:48.394357 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
I20260923 23:05:48.395246 13133 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34937
I20260923 23:05:48.395468 13133 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:48.395695 13133 heartbeater.cc:511] Master 127.5.74.62:34937 requested a full tablet report, sending...
I20260923 23:05:48.396404 12929 ts_manager.cc:194] Registered new tserver with Master: 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301)
I20260923 23:05:48.397275 12929 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:59405
I20260923 23:05:48.401015 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 13002
I20260923 23:05:48.401160 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:48.401412 5416 external_mini_cluster.cc:1428] Setting key f47f259dfa40acd0f9caa882c4deb3ea
I20260923 23:05:48.403942 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.396590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:34649 (local address 127.5.74.62:34937)
0923 23:05:48.396634 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.396636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.396877 (+ 241us) server_negotiation.cc:407] Connection header received
0923 23:05:48.396926 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.396927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.396938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.397010 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:48.397351 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.397990 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.398988 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.399203 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.399267 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.400418 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:48.400419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:48.400419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:48.400426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:48.402655 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.403078 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.403080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.403080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.403141 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.403375 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.403376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.403376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.403496 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:48.403575 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.403760 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.403870 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:05:48.403992 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.396531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34937 (local address 127.5.74.1:34649)
0923 23:05:48.396787 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:48.396805 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:48.396868 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:48.397014 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:48.397014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:48.397201 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:48.397268 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.397270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.398196 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.398197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.398888 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.398891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.399294 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.399295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.399368 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.399902 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:48.399912 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:48.400312 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:48.402790 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.402791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.402792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.402984 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.403218 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.403219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.403219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.403296 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.403565 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:48.403566 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:48.403617 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 23:05:48.403700 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 23:05:48.403845 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab.
Entry for principal oryx/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab.
I20260923 23:05:48.438174 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34937
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab
--principal=oryx/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:48.526093 13142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:48.526324 13142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:48.526351 13142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:48.527657 13142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:48.527702 13142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:48.527726 13142 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:05:48.527745 13142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:48.529394 13142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab
--principal=oryx/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.74.62:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:48.529677 13142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:48.529929 13142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:48.535542 13147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:48.535687 13142 server_base.cc:1034] running on GCE node
W20260923 23:05:48.535561 13148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:48.535631 13150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:48.536078 13142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:48.536379 13142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:48.537560 13142 hybrid_clock.cc:648] HybridClock initialized: now 1790204748537545 us; error 27 us; skew 500 ppm
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:48.540491 13142 init.cc:377] Logged in from keytab as oryx/127.5.74.2@KRBTEST.COM (short username oryx)
I20260923 23:05:48.540988 13142 webserver.cc:460] Webserver started at http://127.5.74.2:33487/ using document root <none> and password file <none>
I20260923 23:05:48.541247 13142 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:48.541294 13142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:48.541438 13142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:48.542615 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "38c89160484e40fa9ffec398b09e0aaa"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "3dc7aa9b063cb31b169e7838b47bcf7a"
server_key_iv: "01b81f596c427f2712c9e9f8593d8278"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.543026 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "38c89160484e40fa9ffec398b09e0aaa"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "3dc7aa9b063cb31b169e7838b47bcf7a"
server_key_iv: "01b81f596c427f2712c9e9f8593d8278"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.544723 13142 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 23:05:48.545631 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:48.545827 13142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:48.545934 13142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "38c89160484e40fa9ffec398b09e0aaa"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "3dc7aa9b063cb31b169e7838b47bcf7a"
server_key_iv: "01b81f596c427f2712c9e9f8593d8278"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.546015 13142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:48.571058 13142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:48.573608 13142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:48.573817 13142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:48.574445 13142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:48.574509 13142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:48.574576 13142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:48.574607 13142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:48.582993 13142 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:37795
I20260923 23:05:48.583055 13272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:37795 every 8 connection(s)
I20260923 23:05:48.583362 13142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:05:48.584764 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 13142
I20260923 23:05:48.584892 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:48.585089 5416 external_mini_cluster.cc:1428] Setting key 17ed80b12c1699313cb452129e51e550
I20260923 23:05:48.587077 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.575008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34937 (local address 127.5.74.2:50447)
0923 23:05:48.575157 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:48.575198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:48.575801 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:48.576201 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:48.576205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:48.576686 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:48.576894 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.576899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.578145 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.578146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.579838 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.579841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.580421 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.580423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.580581 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.581205 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:48.581215 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:48.583157 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:48.584994 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.585006 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.585009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.585408 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.585884 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.585885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.585885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.586041 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.586408 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:48.586414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:48.586643 (+ 229us) client_negotiation.cc:769] Sending connection context
0923 23:05:48.586758 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 23:05:48.586904 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260923 23:05:48.587224 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.575048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:50447 (local address 127.5.74.62:34937)
0923 23:05:48.575140 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.575143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.575341 (+ 198us) server_negotiation.cc:407] Connection header received
0923 23:05:48.575918 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.575920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.575939 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.576056 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:48.577025 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.578033 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.579942 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.580292 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.580398 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.583272 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:48.583273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:48.583274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:48.583282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:48.584885 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.585522 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.585523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.585523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.585590 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.586117 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.586118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.586118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.586298 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:48.586387 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.586807 (+ 420us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.586905 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 23:05:48.587705 13273 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34937
I20260923 23:05:48.587913 13273 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:48.588186 13273 heartbeater.cc:511] Master 127.5.74.62:34937 requested a full tablet report, sending...
I20260923 23:05:48.588701 12929 ts_manager.cc:194] Registered new tserver with Master: 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795)
I20260923 23:05:48.589232 12929 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:50447
WARNING: no policy specified for oryx/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.3@KRBTEST.COM" created.
I20260923 23:05:48.599627 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.589792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:50357 (local address 127.5.74.62:34937)
0923 23:05:48.589846 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.589848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.592616 (+ 2768us) server_negotiation.cc:407] Connection header received
0923 23:05:48.592647 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.592649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.592664 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.592728 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:48.593255 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.593931 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.595210 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.595590 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.595680 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.596788 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:48.596790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:48.596790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:48.596798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:48.598324 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.598701 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.598702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.598703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.598761 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.599013 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.599014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.599015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.599178 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:48.599255 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.599461 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.599534 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:05:48.599637 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.589352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34937 (local address 127.5.74.2:50357)
0923 23:05:48.592318 (+ 2966us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:48.592335 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:48.592445 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:48.592775 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:48.592777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:48.593048 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:48.593142 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.593144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.594058 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.594060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.595071 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.595075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.595680 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.595682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.595762 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.596349 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:48.596354 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:48.596657 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:48.598422 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.598424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.598424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.598610 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.598838 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.598839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.598839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.598916 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.599257 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:48.599258 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:48.599313 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 23:05:48.599390 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 23:05:48.599516 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2921,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab.
Entry for principal oryx/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab.
I20260923 23:05:48.620296 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34937
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab
--principal=oryx/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:48.707509 13282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:48.707712 13282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:48.707736 13282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:48.709000 13282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:48.709040 13282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:48.709061 13282 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:05:48.709079 13282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:48.710587 13282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab
--principal=oryx/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.74.62:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:48.710834 13282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:48.711076 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:48.716286 13290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:48.716398 13282 server_base.cc:1034] running on GCE node
W20260923 23:05:48.716243 13288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:48.716545 13287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:48.716797 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:48.717053 13282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:48.718232 13282 hybrid_clock.cc:648] HybridClock initialized: now 1790204748718220 us; error 30 us; skew 500 ppm
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:48.720957 13282 init.cc:377] Logged in from keytab as oryx/127.5.74.3@KRBTEST.COM (short username oryx)
I20260923 23:05:48.721417 13282 webserver.cc:460] Webserver started at http://127.5.74.3:32865/ using document root <none> and password file <none>
I20260923 23:05:48.721647 13282 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:48.721717 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:48.721872 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:48.722963 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "f361a7f9a49f446f9f7f52ccd34052a0"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "8a22de693d07768796d9578b6fa942a0"
server_key_iv: "5a2be2ef35b68229c3c3a1b30f7fd45e"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.723351 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "f361a7f9a49f446f9f7f52ccd34052a0"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "8a22de693d07768796d9578b6fa942a0"
server_key_iv: "5a2be2ef35b68229c3c3a1b30f7fd45e"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.724915 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 23:05:48.725977 13297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:48.726236 13282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:48.726351 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "f361a7f9a49f446f9f7f52ccd34052a0"
format_stamp: "Formatted at 2026-09-23 23:05:48 on dist-test-slave-j4c5"
server_key: "8a22de693d07768796d9578b6fa942a0"
server_key_iv: "5a2be2ef35b68229c3c3a1b30f7fd45e"
server_key_version: "encryptionkey@0"
I20260923 23:05:48.726438 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:48.749894 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:48.752614 13282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:48.752830 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:48.753494 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:48.753559 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:48.753605 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:48.753633 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:48.761996 13282 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:35139
I20260923 23:05:48.762030 13412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:35139 every 8 connection(s)
I20260923 23:05:48.762449 13282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:05:48.766016 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.754580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:49497 (local address 127.5.74.62:34937)
0923 23:05:48.754654 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.754657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.754821 (+ 164us) server_negotiation.cc:407] Connection header received
0923 23:05:48.755560 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.755562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.755576 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.755649 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:48.756739 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.757241 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.759033 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.759314 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.759481 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.762387 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:48.762389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:48.762389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:48.762397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:48.764215 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.764908 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.764909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.764910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.764972 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.765351 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.765352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.765352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.765482 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:48.765536 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.765871 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.765947 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:05:48.766136 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.754028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34937 (local address 127.5.74.3:49497)
0923 23:05:48.754695 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:48.754739 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:48.755450 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:48.755774 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:48.755779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:48.756253 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:48.756594 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.756602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.757379 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.757380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.758920 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.758923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.759541 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.759543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.759722 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.760648 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:48.760658 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:48.762254 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:48.764339 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.764349 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.764351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.764639 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.765060 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.765060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.765061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.765175 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.765542 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:48.765546 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:48.765674 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 23:05:48.765763 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 23:05:48.765931 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":96,"threads_started":1}
I20260923 23:05:48.766337 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 13282
I20260923 23:05:48.766465 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:48.766654 5416 external_mini_cluster.cc:1428] Setting key a008f443172d5cadbcf37da14583688a
I20260923 23:05:48.766655 13413 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34937
I20260923 23:05:48.766755 13413 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:48.767091 13413 heartbeater.cc:511] Master 127.5.74.62:34937 requested a full tablet report, sending...
I20260923 23:05:48.767644 12929 ts_manager.cc:194] Registered new tserver with Master: f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139)
I20260923 23:05:48.768082 12929 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:49497
I20260923 23:05:48.774251 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.767747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34937 (local address 127.5.74.3:45853)
0923 23:05:48.768274 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:48.768286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:48.768341 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:48.768475 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:48.768476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:48.768697 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:48.768757 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.768759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.769464 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.769465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.769988 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.769990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.770328 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.770329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.770377 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.770837 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:48.770841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:48.771158 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:48.773005 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.773006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.773007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.773259 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.773573 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:48.773574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:48.773575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:48.773659 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:48.773924 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:48.773925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:48.773967 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 23:05:48.774065 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 23:05:48.774156 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":59,"threads_started":1}
I20260923 23:05:48.774325 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.767815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:45853 (local address 127.5.74.62:34937)
0923 23:05:48.767853 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.767855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.768326 (+ 471us) server_negotiation.cc:407] Connection header received
0923 23:05:48.768405 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.768406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.768416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.768470 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:48.768860 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.769350 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.770070 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.770259 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.770316 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.771254 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:48.771256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:48.771256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:48.771262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:48.772919 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.773385 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.773387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.773387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.773465 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.773730 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.773731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.773731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.773847 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:48.773906 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.774167 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.774249 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:05:48.778580 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 23:05:48.802922 12998 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:48.796663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.5.74.62:34937)
0923 23:05:48.796715 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.796717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.796728 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:48.796795 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.796795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.796806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.796865 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:48.797467 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.797963 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.798723 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.798923 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.799026 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.802830 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40932: BlockingRecv error: recv got EOF from 127.0.0.1:40932 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 23 23:05:48 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:48.811699 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.803976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.5.74.62:34937)
0923 23:05:48.804043 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.804045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.804052 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:05:48.804074 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.804075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.804083 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.804149 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:48.804653 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.805180 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.805899 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.806093 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.806180 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.808686 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:48.808688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:48.808688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:48.808696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:48.810133 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.810790 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.810792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.810792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.810859 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:48.811080 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:48.811081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:48.811082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:48.811218 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:48.811349 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.811492 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.811583 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 23:05:48.814384 12929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:48.816054 12929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:48.823740 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.819151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:37795 (local address 127.0.0.1:38662)
0923 23:05:48.819534 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:48.819546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:48.819605 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:48.820227 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:48.820227 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:48.820230 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:48.820273 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.820275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.821597 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.821598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.822599 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.822602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.823295 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.823296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.823399 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.823421 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 23:05:48.823557 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 23:05:48.823628 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":4,"thread_start_us":111,"threads_started":1}
I20260923 23:05:48.823836 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.819113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:35139 (local address 127.0.0.1:55186)
0923 23:05:48.819344 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:48.819393 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:48.819481 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:48.819810 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:48.819812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:48.819814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:48.819889 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.819892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.821427 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.821427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.822521 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.822522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.823349 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.823350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.823432 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.823440 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 23:05:48.823525 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 23:05:48.823597 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":41,"thread_start_us":66,"threads_started":1}
I20260923 23:05:48.823734 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.819231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.5.74.3:35139)
0923 23:05:48.819407 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.819413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.819477 (+ 64us) server_negotiation.cc:407] Connection header received
0923 23:05:48.819622 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.819626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.819643 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.819703 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:48.819985 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.821332 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.822613 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.823217 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.823322 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.823408 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.823556 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.823600 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260923 23:05:48.824431 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.819429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.5.74.2:37795)
0923 23:05:48.819832 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.819836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.819844 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:48.819876 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.819878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.819890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.819999 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:48.820372 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.821507 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.822695 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.823203 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.823268 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.823371 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.824248 (+ 877us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.824301 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":83,"threads_started":1}
I20260923 23:05:48.824862 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.819337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:39301 (local address 127.0.0.1:36500)
0923 23:05:48.819596 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:48.819606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:48.819650 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:48.820082 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:48.820083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:48.820085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:48.820142 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.820144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.821148 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.821150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.822514 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:48.822518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.824498 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.824500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:48.824730 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.824735 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:48.824770 (+ 35us) client_negotiation.cc:240] Negotiation successful
0923 23:05:48.824807 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260923 23:05:48.825012 13207 tablet_service.cc:1467] Processing CreateTablet for tablet c2ee5f196ae04d32abab3a1a71b65981 (DEFAULT_TABLE table=test-table [id=d13259b9a2834e398c95b201f5d41001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:48.824998 13347 tablet_service.cc:1467] Processing CreateTablet for tablet c2ee5f196ae04d32abab3a1a71b65981 (DEFAULT_TABLE table=test-table [id=d13259b9a2834e398c95b201f5d41001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:48.825250 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:48.819631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.5.74.1:39301)
0923 23:05:48.819797 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:48.819836 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:48.819845 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:48.819881 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:48.819883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:48.819898 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:48.820072 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:48.820283 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.821041 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.823468 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:48.824341 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:48.825009 (+ 668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:48.825107 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:48.825133 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 23:05:48.825170 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260923 23:05:48.825397 13347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ee5f196ae04d32abab3a1a71b65981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:48.825397 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ee5f196ae04d32abab3a1a71b65981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:48.825856 13067 tablet_service.cc:1467] Processing CreateTablet for tablet c2ee5f196ae04d32abab3a1a71b65981 (DEFAULT_TABLE table=test-table [id=d13259b9a2834e398c95b201f5d41001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:48.826128 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ee5f196ae04d32abab3a1a71b65981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:48.829672 13441 tablet_bootstrap.cc:492] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: Bootstrap starting.
I20260923 23:05:48.829994 13442 tablet_bootstrap.cc:492] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: Bootstrap starting.
I20260923 23:05:48.830070 13443 tablet_bootstrap.cc:492] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: Bootstrap starting.
I20260923 23:05:48.830614 13441 tablet_bootstrap.cc:654] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:48.830736 13443 tablet_bootstrap.cc:654] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:48.830780 13442 tablet_bootstrap.cc:654] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:48.831028 13441 log.cc:824] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:48.831105 13442 log.cc:824] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:48.831106 13443 log.cc:824] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:48.832113 13441 tablet_bootstrap.cc:492] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: No bootstrap required, opened a new log
I20260923 23:05:48.832214 13441 ts_tablet_manager.cc:1397] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260923 23:05:48.832512 13442 tablet_bootstrap.cc:492] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: No bootstrap required, opened a new log
I20260923 23:05:48.832599 13442 ts_tablet_manager.cc:1397] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:05:48.832650 13443 tablet_bootstrap.cc:492] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: No bootstrap required, opened a new log
I20260923 23:05:48.832718 13443 ts_tablet_manager.cc:1397] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:05:48.834381 13441 raft_consensus.cc:348] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } }
I20260923 23:05:48.834584 13441 raft_consensus.cc:374] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:48.834621 13441 raft_consensus.cc:729] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f361a7f9a49f446f9f7f52ccd34052a0, State: Initialized, Role: FOLLOWER
I20260923 23:05:48.834501 13442 raft_consensus.cc:348] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } }
I20260923 23:05:48.834671 13442 raft_consensus.cc:374] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:48.834704 13442 raft_consensus.cc:729] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c89160484e40fa9ffec398b09e0aaa, State: Initialized, Role: FOLLOWER
I20260923 23:05:48.834779 13441 consensus_queue.cc:260] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } }
I20260923 23:05:48.834836 13442 consensus_queue.cc:260] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } }
I20260923 23:05:48.834794 13443 raft_consensus.cc:348] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } }
I20260923 23:05:48.834954 13443 raft_consensus.cc:374] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:48.834997 13443 raft_consensus.cc:729] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb06458ffec40c897035e161d13175b, State: Initialized, Role: FOLLOWER
I20260923 23:05:48.835111 13441 ts_tablet_manager.cc:1428] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 23:05:48.835134 13443 consensus_queue.cc:260] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } }
I20260923 23:05:48.835197 13442 ts_tablet_manager.cc:1428] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:05:48.835237 13413 heartbeater.cc:503] Master 127.5.74.62:34937 was elected leader, sending a full tablet report...
I20260923 23:05:48.835373 13443 ts_tablet_manager.cc:1428] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:05:48.835526 13273 heartbeater.cc:503] Master 127.5.74.62:34937 was elected leader, sending a full tablet report...
I20260923 23:05:48.835536 13133 heartbeater.cc:503] Master 127.5.74.62:34937 was elected leader, sending a full tablet report...
W20260923 23:05:48.892437 13134 tablet.cc:2369] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:05:49.013119 13414 tablet.cc:2369] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:49.019071 13449 raft_consensus.cc:482] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:49.019225 13449 raft_consensus.cc:504] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } }
I20260923 23:05:49.019650 13449 leader_election.cc:288] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139), 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795)
I20260923 23:05:49.022827 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.020012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:56253 (local address 127.5.74.2:37795)
0923 23:05:49.020055 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.020058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.020138 (+ 80us) server_negotiation.cc:407] Connection header received
0923 23:05:49.020222 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.020223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.020235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.020307 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:49.020512 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.020997 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.022073 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.022447 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.022536 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.022547 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.022698 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.022738 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:05:49.022843 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.020059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:57631 (local address 127.5.74.3:35139)
0923 23:05:49.020107 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.020109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.020119 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:49.020214 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.020215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.020232 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.020347 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:49.020514 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.021030 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.022074 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.022469 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.022558 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.022569 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.022720 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.022760 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:05:49.022882 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.019901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:35139 (local address 127.5.74.1:57631)
0923 23:05:49.020044 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:49.020056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:49.020115 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:49.020351 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:49.020351 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:49.020354 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:49.020437 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.020438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.021125 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.021126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.021960 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.021962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.022559 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.022560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.022640 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.022643 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:49.022712 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 23:05:49.022757 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":74,"thread_start_us":61,"threads_started":1}
I20260923 23:05:49.022873 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.019901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:37795 (local address 127.5.74.1:56253)
0923 23:05:49.020031 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:49.020045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:49.020138 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:49.020327 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:49.020328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:49.020330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:49.020430 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.020432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.021094 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.021095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.021961 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.021963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.022538 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.022539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.022616 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.022620 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:49.022691 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 23:05:49.022739 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":30,"thread_start_us":65,"threads_started":1}
I20260923 23:05:49.023259 13367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ee5f196ae04d32abab3a1a71b65981" candidate_uuid: "6fb06458ffec40c897035e161d13175b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" is_pre_election: true
I20260923 23:05:49.023402 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ee5f196ae04d32abab3a1a71b65981" candidate_uuid: "6fb06458ffec40c897035e161d13175b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c89160484e40fa9ffec398b09e0aaa" is_pre_election: true
I20260923 23:05:49.023475 13367 raft_consensus.cc:2454] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb06458ffec40c897035e161d13175b in term 0.
I20260923 23:05:49.023576 13227 raft_consensus.cc:2454] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb06458ffec40c897035e161d13175b in term 0.
I20260923 23:05:49.023747 13020 leader_election.cc:302] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb06458ffec40c897035e161d13175b, f361a7f9a49f446f9f7f52ccd34052a0; no voters:
I20260923 23:05:49.023939 13449 raft_consensus.cc:2781] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:49.024035 13449 raft_consensus.cc:482] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:49.024075 13449 raft_consensus.cc:3037] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.024847 13449 raft_consensus.cc:504] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } }
I20260923 23:05:49.025015 13449 leader_election.cc:288] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [CANDIDATE]: Term 1 election: Requested vote from peers f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139), 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795)
I20260923 23:05:49.025185 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ee5f196ae04d32abab3a1a71b65981" candidate_uuid: "6fb06458ffec40c897035e161d13175b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c89160484e40fa9ffec398b09e0aaa"
I20260923 23:05:49.025228 13367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ee5f196ae04d32abab3a1a71b65981" candidate_uuid: "6fb06458ffec40c897035e161d13175b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f361a7f9a49f446f9f7f52ccd34052a0"
I20260923 23:05:49.025281 13227 raft_consensus.cc:3037] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.025332 13367 raft_consensus.cc:3037] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.026110 13367 raft_consensus.cc:2454] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb06458ffec40c897035e161d13175b in term 1.
I20260923 23:05:49.026290 13020 leader_election.cc:302] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb06458ffec40c897035e161d13175b, f361a7f9a49f446f9f7f52ccd34052a0; no voters:
I20260923 23:05:49.026404 13449 raft_consensus.cc:2781] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:49.026551 13449 raft_consensus.cc:686] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 1 LEADER]: Becoming Leader. State: Replica: 6fb06458ffec40c897035e161d13175b, State: Running, Role: LEADER
I20260923 23:05:49.026566 13227 raft_consensus.cc:2454] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb06458ffec40c897035e161d13175b in term 1.
I20260923 23:05:49.026675 13449 consensus_queue.cc:237] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } }
I20260923 23:05:49.027709 12930 catalog_manager.cc:5495] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fb06458ffec40c897035e161d13175b (127.5.74.1). New cstate: current_term: 1 leader_uuid: "6fb06458ffec40c897035e161d13175b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } health_report { overall_health: HEALTHY } } }
I20260923 23:05:49.060315 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.057297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.5.74.1:39301)
0923 23:05:49.057463 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.057465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.057590 (+ 125us) server_negotiation.cc:407] Connection header received
0923 23:05:49.057651 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.057651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.057674 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.057752 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:49.057972 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.058591 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.059490 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.059737 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.059801 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.059960 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:49.060065 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:49.060129 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.060231 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.060252 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260923 23:05:49.062495 13227 raft_consensus.cc:1261] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Refusing update from remote peer 6fb06458ffec40c897035e161d13175b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:49.062826 13367 raft_consensus.cc:1261] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Refusing update from remote peer 6fb06458ffec40c897035e161d13175b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:49.062891 13452 consensus_queue.cc:1035] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:49.063143 13452 consensus_queue.cc:1035] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:49.065827 13455 mvcc.cc:205] Tried to move back new op lower bound from 7332678652156362752 to 7332678652013920256. Current Snapshot: MvccSnapshot[applied={T|T < 7332678652156362752}]
I20260923 23:05:49.066236 13457 mvcc.cc:205] Tried to move back new op lower bound from 7332678652156362752 to 7332678652013920256. Current Snapshot: MvccSnapshot[applied={T|T < 7332678652156362752}]
I20260923 23:05:49.066529 13456 mvcc.cc:205] Tried to move back new op lower bound from 7332678652156362752 to 7332678652013920256. Current Snapshot: MvccSnapshot[applied={T|T < 7332678652156362752}]
I20260923 23:05:49.069659 12929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:49.069801 12929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:49.070586 12929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879: Sending DeleteTablet for 3 replicas of tablet c2ee5f196ae04d32abab3a1a71b65981
I20260923 23:05:49.071038 13347 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ee5f196ae04d32abab3a1a71b65981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:49 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:55186
I20260923 23:05:49.071177 13207 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ee5f196ae04d32abab3a1a71b65981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:49 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:38662
I20260923 23:05:49.071285 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ee5f196ae04d32abab3a1a71b65981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:49 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:36500
I20260923 23:05:49.071238 12929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:49.071594 12929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:49.071610 13465 tablet_replica.cc:330] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: stopping tablet replica
I20260923 23:05:49.071724 13465 raft_consensus.cc:2229] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:49.071856 13465 raft_consensus.cc:2258] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:49.072017 13466 tablet_replica.cc:330] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: stopping tablet replica
I20260923 23:05:49.072100 13466 raft_consensus.cc:2229] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:49.072283 13466 raft_consensus.cc:2258] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:49.072600 13465 ts_tablet_manager.cc:1905] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:49.072696 13466 ts_tablet_manager.cc:1905] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:49.073846 13347 tablet_service.cc:1467] Processing CreateTablet for tablet 314686ff47aa4e22963e99cb4e531f45 (DEFAULT_TABLE table=test-table [id=1bcb04fbaa6043d28f6958878bf0396a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:49.073987 13347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314686ff47aa4e22963e99cb4e531f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:49.074718 13466 ts_tablet_manager.cc:1918] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:49.074796 13466 log.cc:1196] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/c2ee5f196ae04d32abab3a1a71b65981
I20260923 23:05:49.075059 13466 ts_tablet_manager.cc:1939] T c2ee5f196ae04d32abab3a1a71b65981 P 6fb06458ffec40c897035e161d13175b: Deleting consensus metadata
I20260923 23:05:49.075050 13465 ts_tablet_manager.cc:1918] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:49.075124 13465 log.cc:1196] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/c2ee5f196ae04d32abab3a1a71b65981
I20260923 23:05:49.075361 13441 tablet_bootstrap.cc:492] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: Bootstrap starting.
I20260923 23:05:49.075641 13207 tablet_service.cc:1467] Processing CreateTablet for tablet 314686ff47aa4e22963e99cb4e531f45 (DEFAULT_TABLE table=test-table [id=1bcb04fbaa6043d28f6958878bf0396a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:49.075771 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314686ff47aa4e22963e99cb4e531f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:49.075834 13441 tablet_bootstrap.cc:654] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:49.075888 12916 catalog_manager.cc:4841] TS 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301): tablet c2ee5f196ae04d32abab3a1a71b65981 (table test-table [id=d13259b9a2834e398c95b201f5d41001]) successfully deleted
I20260923 23:05:49.076303 13465 ts_tablet_manager.cc:1939] T c2ee5f196ae04d32abab3a1a71b65981 P 38c89160484e40fa9ffec398b09e0aaa: Deleting consensus metadata
I20260923 23:05:49.076637 13464 tablet_replica.cc:330] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: stopping tablet replica
I20260923 23:05:49.076730 12914 catalog_manager.cc:4841] TS 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795): tablet c2ee5f196ae04d32abab3a1a71b65981 (table test-table [id=d13259b9a2834e398c95b201f5d41001]) successfully deleted
I20260923 23:05:49.076918 13464 raft_consensus.cc:2229] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:49.077075 13464 raft_consensus.cc:2258] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:49.077411 13067 tablet_service.cc:1467] Processing CreateTablet for tablet 314686ff47aa4e22963e99cb4e531f45 (DEFAULT_TABLE table=test-table [id=1bcb04fbaa6043d28f6958878bf0396a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:49.077535 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314686ff47aa4e22963e99cb4e531f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:49.077739 13464 ts_tablet_manager.cc:1905] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:49.079030 13443 tablet_bootstrap.cc:492] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: Bootstrap starting.
I20260923 23:05:49.079036 13442 tablet_bootstrap.cc:492] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: Bootstrap starting.
I20260923 23:05:49.079417 13443 tablet_bootstrap.cc:654] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:49.079463 13442 tablet_bootstrap.cc:654] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:49.079646 13464 ts_tablet_manager.cc:1918] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:49.079764 13464 log.cc:1196] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/c2ee5f196ae04d32abab3a1a71b65981
I20260923 23:05:49.080225 13464 ts_tablet_manager.cc:1939] T c2ee5f196ae04d32abab3a1a71b65981 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting consensus metadata
I20260923 23:05:49.080341 13442 tablet_bootstrap.cc:492] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: No bootstrap required, opened a new log
I20260923 23:05:49.080400 13442 ts_tablet_manager.cc:1397] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:49.080441 13443 tablet_bootstrap.cc:492] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: No bootstrap required, opened a new log
I20260923 23:05:49.080502 13443 ts_tablet_manager.cc:1397] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:49.080562 13442 raft_consensus.cc:348] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.080644 13442 raft_consensus.cc:374] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:49.080670 13442 raft_consensus.cc:729] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c89160484e40fa9ffec398b09e0aaa, State: Initialized, Role: FOLLOWER
I20260923 23:05:49.080648 13443 raft_consensus.cc:348] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.080727 13443 raft_consensus.cc:374] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:49.080714 13442 consensus_queue.cc:260] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.080754 13443 raft_consensus.cc:729] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb06458ffec40c897035e161d13175b, State: Initialized, Role: FOLLOWER
I20260923 23:05:49.080799 13442 ts_tablet_manager.cc:1428] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:49.080807 13443 consensus_queue.cc:260] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.080884 13443 ts_tablet_manager.cc:1428] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:49.081302 13441 tablet_bootstrap.cc:492] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: No bootstrap required, opened a new log
I20260923 23:05:49.081373 13441 ts_tablet_manager.cc:1397] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260923 23:05:49.081526 13441 raft_consensus.cc:348] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.081607 13441 raft_consensus.cc:374] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:49.081622 13441 raft_consensus.cc:729] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f361a7f9a49f446f9f7f52ccd34052a0, State: Initialized, Role: FOLLOWER
I20260923 23:05:49.081681 13441 consensus_queue.cc:260] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.081771 13441 ts_tablet_manager.cc:1428] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:49.082683 12916 catalog_manager.cc:4841] TS f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139): tablet c2ee5f196ae04d32abab3a1a71b65981 (table test-table [id=d13259b9a2834e398c95b201f5d41001]) successfully deleted
W20260923 23:05:49.084210 13274 tablet.cc:2369] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:05:49.370002 13448 raft_consensus.cc:482] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:49.370139 13448 raft_consensus.cc:504] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.370537 13448 leader_election.cc:288] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301), f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139)
I20260923 23:05:49.373659 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.371047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:47095 (local address 127.5.74.3:35139)
0923 23:05:49.371127 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.371130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.371139 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:49.371169 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.371170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.371182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.371256 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:49.371456 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.371938 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.372937 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.373300 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.373387 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.373397 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.373545 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.373584 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 23:05:49.373669 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.370860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:53791 (local address 127.5.74.1:39301)
0923 23:05:49.370929 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.370934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.370964 (+ 30us) server_negotiation.cc:407] Connection header received
0923 23:05:49.371019 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.371020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.371030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.371116 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:49.371354 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.371860 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.372927 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.373317 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.373401 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.373411 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.373562 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.373603 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 23:05:49.373685 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.370764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:39301 (local address 127.5.74.2:53791)
0923 23:05:49.370889 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:49.370901 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:49.370958 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:49.371143 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:49.371144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:49.371146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:49.371218 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.371219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.371951 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.371952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.372833 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.372835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.373402 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.373402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.373477 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.373481 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:49.373543 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 23:05:49.373591 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":18,"thread_start_us":79,"threads_started":1}
I20260923 23:05:49.373692 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.370970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:35139 (local address 127.5.74.2:47095)
0923 23:05:49.371065 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:49.371073 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:49.371114 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:49.371260 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:49.371261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:49.371263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:49.371324 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.371326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.372026 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.372027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.372842 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.372844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.373388 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.373389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.373466 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.373470 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:49.373539 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 23:05:49.373584 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":21,"thread_start_us":42,"threads_started":1}
I20260923 23:05:49.373974 13367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314686ff47aa4e22963e99cb4e531f45" candidate_uuid: "38c89160484e40fa9ffec398b09e0aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" is_pre_election: true
I20260923 23:05:49.374069 13367 raft_consensus.cc:2454] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c89160484e40fa9ffec398b09e0aaa in term 0.
I20260923 23:05:49.374044 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314686ff47aa4e22963e99cb4e531f45" candidate_uuid: "38c89160484e40fa9ffec398b09e0aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb06458ffec40c897035e161d13175b" is_pre_election: true
I20260923 23:05:49.374226 13087 raft_consensus.cc:2454] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c89160484e40fa9ffec398b09e0aaa in term 0.
I20260923 23:05:49.374240 13160 leader_election.cc:302] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c89160484e40fa9ffec398b09e0aaa, f361a7f9a49f446f9f7f52ccd34052a0; no voters:
I20260923 23:05:49.374382 13448 raft_consensus.cc:2781] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:49.374437 13448 raft_consensus.cc:482] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:49.374465 13448 raft_consensus.cc:3037] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.375134 13448 raft_consensus.cc:504] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.375283 13448 leader_election.cc:288] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [CANDIDATE]: Term 1 election: Requested vote from peers 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301), f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139)
I20260923 23:05:49.375505 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314686ff47aa4e22963e99cb4e531f45" candidate_uuid: "38c89160484e40fa9ffec398b09e0aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb06458ffec40c897035e161d13175b"
I20260923 23:05:49.375604 13087 raft_consensus.cc:3037] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.375572 13367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314686ff47aa4e22963e99cb4e531f45" candidate_uuid: "38c89160484e40fa9ffec398b09e0aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f361a7f9a49f446f9f7f52ccd34052a0"
I20260923 23:05:49.375674 13367 raft_consensus.cc:3037] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.376333 13367 raft_consensus.cc:2454] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c89160484e40fa9ffec398b09e0aaa in term 1.
I20260923 23:05:49.376423 13087 raft_consensus.cc:2454] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c89160484e40fa9ffec398b09e0aaa in term 1.
I20260923 23:05:49.376492 13160 leader_election.cc:302] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c89160484e40fa9ffec398b09e0aaa, f361a7f9a49f446f9f7f52ccd34052a0; no voters:
I20260923 23:05:49.376597 13448 raft_consensus.cc:2781] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:49.376714 13448 raft_consensus.cc:686] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 1 LEADER]: Becoming Leader. State: Replica: 38c89160484e40fa9ffec398b09e0aaa, State: Running, Role: LEADER
I20260923 23:05:49.376784 13448 consensus_queue.cc:237] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.377485 12929 catalog_manager.cc:5495] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2). New cstate: current_term: 1 leader_uuid: "38c89160484e40fa9ffec398b09e0aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } health_report { overall_health: UNKNOWN } } }
Sep 23 23:05:49 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204748, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.62@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:49.457993 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.449028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34937 (local address 127.0.0.1:54706)
0923 23:05:49.449209 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:49.449224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:49.449305 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:49.449618 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:49.449619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:49.449943 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:49.449989 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.449991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.450706 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.450715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.451349 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.451352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.451739 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.451740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.451852 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.452559 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:49.452563 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:49.454102 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:49.456677 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:49.456686 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:49.456688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:49.456921 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:49.457124 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:49.457124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:49.457125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:49.457236 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:49.457584 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:49.457591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:49.457651 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 23:05:49.457719 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 23:05:49.457875 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260923 23:05:49.458151 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.449157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.5.74.62:34937)
0923 23:05:49.449441 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.449443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.449453 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:49.449493 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.449494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.449503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.449570 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:49.450072 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.450612 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.451448 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.451636 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.451697 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.454305 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:49.454307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:49.454307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:49.454314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:49.456558 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:49.456992 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:49.456993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:49.456994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:49.457049 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:49.457336 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:49.457337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:49.457337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:49.457489 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:49.457560 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.457788 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.457969 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":257,"thread_start_us":166,"threads_started":1}
I20260923 23:05:49.460602 12929 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:54706:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 23:05:49.460811 12929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:49.462402 13207 tablet_service.cc:1467] Processing CreateTablet for tablet 69fa3c1eafc54910b7c183492b4fae04 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24005ead332e439b90b0658fb12b1419]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:05:49.462484 13067 tablet_service.cc:1467] Processing CreateTablet for tablet 69fa3c1eafc54910b7c183492b4fae04 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24005ead332e439b90b0658fb12b1419]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:05:49.462529 13347 tablet_service.cc:1467] Processing CreateTablet for tablet 69fa3c1eafc54910b7c183492b4fae04 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24005ead332e439b90b0658fb12b1419]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:05:49.462586 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69fa3c1eafc54910b7c183492b4fae04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:49.462598 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69fa3c1eafc54910b7c183492b4fae04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:49.462657 13347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69fa3c1eafc54910b7c183492b4fae04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:49.464279 13442 tablet_bootstrap.cc:492] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: Bootstrap starting.
I20260923 23:05:49.464334 13441 tablet_bootstrap.cc:492] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: Bootstrap starting.
I20260923 23:05:49.464769 13442 tablet_bootstrap.cc:654] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:49.464769 13441 tablet_bootstrap.cc:654] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:49.465152 13443 tablet_bootstrap.cc:492] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: Bootstrap starting.
I20260923 23:05:49.465481 13443 tablet_bootstrap.cc:654] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:49.465507 13442 tablet_bootstrap.cc:492] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: No bootstrap required, opened a new log
I20260923 23:05:49.465560 13442 ts_tablet_manager.cc:1397] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:49.465646 13441 tablet_bootstrap.cc:492] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: No bootstrap required, opened a new log
I20260923 23:05:49.465700 13441 ts_tablet_manager.cc:1397] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:05:49.465728 13442 raft_consensus.cc:348] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.465847 13442 raft_consensus.cc:374] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:49.465871 13442 raft_consensus.cc:729] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c89160484e40fa9ffec398b09e0aaa, State: Initialized, Role: FOLLOWER
I20260923 23:05:49.465853 13441 raft_consensus.cc:348] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.465927 13441 raft_consensus.cc:374] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:49.465924 13442 consensus_queue.cc:260] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.465960 13441 raft_consensus.cc:729] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f361a7f9a49f446f9f7f52ccd34052a0, State: Initialized, Role: FOLLOWER
I20260923 23:05:49.466017 13442 ts_tablet_manager.cc:1428] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:49.466006 13441 consensus_queue.cc:260] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.466114 13441 ts_tablet_manager.cc:1428] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:49.466120 13448 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.466209 13448 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:49.466259 13447 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.466341 13447 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:49.466861 13443 tablet_bootstrap.cc:492] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: No bootstrap required, opened a new log
I20260923 23:05:49.466935 13443 ts_tablet_manager.cc:1397] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:49.467058 13443 raft_consensus.cc:348] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.467139 13443 raft_consensus.cc:374] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:49.467187 13443 raft_consensus.cc:729] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb06458ffec40c897035e161d13175b, State: Initialized, Role: FOLLOWER
I20260923 23:05:49.467233 13443 consensus_queue.cc:260] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.467340 13443 ts_tablet_manager.cc:1428] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:49.467422 13452 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.467525 13452 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:49.711083 13452 raft_consensus.cc:482] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:49.711213 13452 raft_consensus.cc:504] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.711549 13452 leader_election.cc:288] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795), f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139)
I20260923 23:05:49.711751 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fa3c1eafc54910b7c183492b4fae04" candidate_uuid: "6fb06458ffec40c897035e161d13175b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c89160484e40fa9ffec398b09e0aaa" is_pre_election: true
I20260923 23:05:49.711905 13227 raft_consensus.cc:2454] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb06458ffec40c897035e161d13175b in term 0.
I20260923 23:05:49.711870 13367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fa3c1eafc54910b7c183492b4fae04" candidate_uuid: "6fb06458ffec40c897035e161d13175b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" is_pre_election: true
I20260923 23:05:49.711974 13367 raft_consensus.cc:2454] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb06458ffec40c897035e161d13175b in term 0.
I20260923 23:05:49.712110 13018 leader_election.cc:302] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c89160484e40fa9ffec398b09e0aaa, 6fb06458ffec40c897035e161d13175b; no voters:
I20260923 23:05:49.712234 13452 raft_consensus.cc:2781] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:49.712311 13452 raft_consensus.cc:482] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:49.712349 13452 raft_consensus.cc:3037] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.713052 13452 raft_consensus.cc:504] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.713207 13452 leader_election.cc:288] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [CANDIDATE]: Term 1 election: Requested vote from peers 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795), f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139)
I20260923 23:05:49.713442 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fa3c1eafc54910b7c183492b4fae04" candidate_uuid: "6fb06458ffec40c897035e161d13175b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c89160484e40fa9ffec398b09e0aaa"
I20260923 23:05:49.713413 13367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fa3c1eafc54910b7c183492b4fae04" candidate_uuid: "6fb06458ffec40c897035e161d13175b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f361a7f9a49f446f9f7f52ccd34052a0"
I20260923 23:05:49.713517 13367 raft_consensus.cc:3037] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.713528 13227 raft_consensus.cc:3037] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.714233 13367 raft_consensus.cc:2454] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb06458ffec40c897035e161d13175b in term 1.
I20260923 23:05:49.714294 13227 raft_consensus.cc:2454] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb06458ffec40c897035e161d13175b in term 1.
I20260923 23:05:49.714452 13020 leader_election.cc:302] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb06458ffec40c897035e161d13175b, f361a7f9a49f446f9f7f52ccd34052a0; no voters:
I20260923 23:05:49.714550 13452 raft_consensus.cc:2781] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:49.714672 13452 raft_consensus.cc:686] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [term 1 LEADER]: Becoming Leader. State: Replica: 6fb06458ffec40c897035e161d13175b, State: Running, Role: LEADER
I20260923 23:05:49.714787 13452 consensus_queue.cc:237] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.715111 13484 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: TxnStatusTablet state changed. Reason: New leader 6fb06458ffec40c897035e161d13175b. Latest consensus state: current_term: 1 leader_uuid: "6fb06458ffec40c897035e161d13175b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.715209 13484 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:05:49.715355 13486 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 23:05:49.715492 13486 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 23:05:49.715487 12929 catalog_manager.cc:5495] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fb06458ffec40c897035e161d13175b (127.5.74.1). New cstate: current_term: 1 leader_uuid: "6fb06458ffec40c897035e161d13175b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:49.754487 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.750965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:39301 (local address 127.0.0.1:50758)
0923 23:05:49.751047 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:49.751060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:49.751133 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:49.751278 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:49.751279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:49.751280 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:49.751348 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.751351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.752289 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.752290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.753203 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.753207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.753723 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.753725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.753800 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.753829 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:49.754289 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:49.754304 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 23:05:49.754371 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:05:49.754418 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 23:05:49.754501 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.751055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.5.74.1:39301)
0923 23:05:49.751105 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.751107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.751117 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:49.751203 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.751204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.751216 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.751271 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:49.751418 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.752202 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.753319 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.753621 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.753702 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.753929 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:49.753967 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:49.754188 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.754398 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.754423 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:05:49.756072 13227 raft_consensus.cc:1261] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Refusing update from remote peer 6fb06458ffec40c897035e161d13175b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:49.756191 13367 raft_consensus.cc:1261] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Refusing update from remote peer 6fb06458ffec40c897035e161d13175b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:49.756369 13452 consensus_queue.cc:1035] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:49.756614 13484 consensus_queue.cc:1035] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:49.757122 13448 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: TxnStatusTablet state changed. Reason: New leader 6fb06458ffec40c897035e161d13175b. Latest consensus state: current_term: 1 leader_uuid: "6fb06458ffec40c897035e161d13175b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.757221 13448 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:49.757261 13447 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: TxnStatusTablet state changed. Reason: New leader 6fb06458ffec40c897035e161d13175b. Latest consensus state: current_term: 1 leader_uuid: "6fb06458ffec40c897035e161d13175b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.757346 13447 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:49.757897 13452 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fb06458ffec40c897035e161d13175b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.758008 13452 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:05:49.758325 13448 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fb06458ffec40c897035e161d13175b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.758416 13448 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P 38c89160484e40fa9ffec398b09e0aaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:49.758774 13484 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fb06458ffec40c897035e161d13175b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.758867 13484 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P 6fb06458ffec40c897035e161d13175b: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:05:49.759073 13447 tablet_replica.cc:439] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fb06458ffec40c897035e161d13175b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } } }
I20260923 23:05:49.759159 13447 tablet_replica.cc:442] T 69fa3c1eafc54910b7c183492b4fae04 P f361a7f9a49f446f9f7f52ccd34052a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:05:49.764515 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.761352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.5.74.2:37795)
0923 23:05:49.761467 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.761469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.761479 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:49.761571 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.761572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.761601 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.761700 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:49.761973 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.762547 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.763426 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.763667 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.763772 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.763960 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:49.764136 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:49.764234 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.764364 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.764388 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
I20260923 23:05:49.770654 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.767744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:39301 (local address 127.5.74.2:42209)
0923 23:05:49.767939 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:49.767949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:49.767999 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:49.768214 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:49.768215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:49.768217 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:49.768269 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.768270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.769148 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.769149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.769728 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.769730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.770251 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.770253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.770307 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.770320 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:49.770501 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:49.770502 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:05:49.770537 (+ 35us) client_negotiation.cc:240] Negotiation successful
0923 23:05:49.770582 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20260923 23:05:49.770641 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.767806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:42209 (local address 127.5.74.1:39301)
0923 23:05:49.767870 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.767872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.768007 (+ 135us) server_negotiation.cc:407] Connection header received
0923 23:05:49.768053 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.768054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.768067 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.768121 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:49.768332 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.769056 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.769818 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.770152 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.770225 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.770406 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:49.770441 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:49.770495 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.770540 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.770560 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:05:49.773150 13367 raft_consensus.cc:1261] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Refusing update from remote peer 38c89160484e40fa9ffec398b09e0aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:49.773260 13087 raft_consensus.cc:1261] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Refusing update from remote peer 38c89160484e40fa9ffec398b09e0aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:49.773406 13448 consensus_queue.cc:1035] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:49.773515 13472 consensus_queue.cc:1035] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:49.782330 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.779387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:34257 (local address 127.5.74.2:37795)
0923 23:05:49.779435 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.779437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.779581 (+ 144us) server_negotiation.cc:407] Connection header received
0923 23:05:49.779670 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.779670 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.779679 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.779717 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:49.779840 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.780408 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.781479 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.781782 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.781824 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.782065 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:49.782101 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:49.782176 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.782242 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.782264 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:05:49.782363 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.779337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:37795 (local address 127.5.74.1:34257)
0923 23:05:49.779513 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:49.779527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:49.779609 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:49.779728 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:49.779729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:49.779730 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:49.779788 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.779789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.780483 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.780484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.781373 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:49.781377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.781884 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.781884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:49.781938 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.781957 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:49.782192 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:49.782194 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:05:49.782241 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:05:49.782284 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260923 23:05:49.792272 12929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:49.792409 12929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:49.793143 12929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879: Sending DeleteTablet for 3 replicas of tablet 314686ff47aa4e22963e99cb4e531f45
I20260923 23:05:49.793447 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet 314686ff47aa4e22963e99cb4e531f45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:49 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:36500
I20260923 23:05:49.793649 13207 tablet_service.cc:1514] Processing DeleteTablet for tablet 314686ff47aa4e22963e99cb4e531f45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:49 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:38662
I20260923 23:05:49.793924 13347 tablet_service.cc:1514] Processing DeleteTablet for tablet 314686ff47aa4e22963e99cb4e531f45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:49 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:55186
I20260923 23:05:49.794165 13511 tablet_replica.cc:330] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: stopping tablet replica
I20260923 23:05:49.794232 13511 raft_consensus.cc:2229] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:49.794277 13511 raft_consensus.cc:2258] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:49.794386 13512 tablet_replica.cc:330] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: stopping tablet replica
I20260923 23:05:49.794451 13512 raft_consensus.cc:2229] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:49.794543 13512 raft_consensus.cc:2258] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:49.794853 13511 ts_tablet_manager.cc:1905] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:49.796320 13512 ts_tablet_manager.cc:1905] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:49.798069 13512 ts_tablet_manager.cc:1918] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:49.798137 13512 log.cc:1196] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/314686ff47aa4e22963e99cb4e531f45
I20260923 23:05:49.798374 13512 ts_tablet_manager.cc:1939] T 314686ff47aa4e22963e99cb4e531f45 P 38c89160484e40fa9ffec398b09e0aaa: Deleting consensus metadata
I20260923 23:05:49.798913 12914 catalog_manager.cc:4841] TS 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795): tablet 314686ff47aa4e22963e99cb4e531f45 (table test-table [id=1bcb04fbaa6043d28f6958878bf0396a]) successfully deleted
I20260923 23:05:49.799228 13511 ts_tablet_manager.cc:1918] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:49.799292 13511 log.cc:1196] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/314686ff47aa4e22963e99cb4e531f45
I20260923 23:05:49.799510 13511 ts_tablet_manager.cc:1939] T 314686ff47aa4e22963e99cb4e531f45 P 6fb06458ffec40c897035e161d13175b: Deleting consensus metadata
I20260923 23:05:49.800009 12916 catalog_manager.cc:4841] TS 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301): tablet 314686ff47aa4e22963e99cb4e531f45 (table test-table [id=1bcb04fbaa6043d28f6958878bf0396a]) successfully deleted
I20260923 23:05:49.804289 13513 tablet_replica.cc:330] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: stopping tablet replica
I20260923 23:05:49.804379 13513 raft_consensus.cc:2229] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:49.804436 13513 raft_consensus.cc:2258] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:49.804726 13513 ts_tablet_manager.cc:1905] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:49.806540 13513 ts_tablet_manager.cc:1918] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:49.806610 13513 log.cc:1196] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/314686ff47aa4e22963e99cb4e531f45
I20260923 23:05:49.806839 13513 ts_tablet_manager.cc:1939] T 314686ff47aa4e22963e99cb4e531f45 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting consensus metadata
I20260923 23:05:49.807214 12916 catalog_manager.cc:4841] TS f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139): tablet 314686ff47aa4e22963e99cb4e531f45 (table test-table [id=1bcb04fbaa6043d28f6958878bf0396a]) successfully deleted
Sep 23 23:05:49 dist-test-slave-j4c5 krb5kdc[12885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:05:49 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:49 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 23:05:49.830948 13480 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:49.823733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.5.74.62:34937)
0923 23:05:49.823784 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.823786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.823957 (+ 171us) server_negotiation.cc:407] Connection header received
0923 23:05:49.824020 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.824021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.824034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.824103 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:49.824872 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.825643 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.826789 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.827091 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.827172 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.830862 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54720: BlockingRecv error: recv got EOF from 127.0.0.1:54720 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 23 23:05:49 dist-test-slave-j4c5 krb5kdc[12885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM
I20260923 23:05:49.842105 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.832589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.5.74.62:34937)
0923 23:05:49.832658 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.832660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.832844 (+ 184us) server_negotiation.cc:407] Connection header received
0923 23:05:49.832916 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.832917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.832929 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.832990 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:49.833678 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.834474 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.835732 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.836040 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.836116 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.838957 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:49.838959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:49.838959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:49.838968 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:49.840745 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:49.841189 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:49.841190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:49.841191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:49.841253 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:49.841473 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:49.841475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:49.841475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:49.841633 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:49.841749 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.841912 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.841999 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:49.843820 12929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:49.843997 12929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:49.845712 13347 tablet_service.cc:1467] Processing CreateTablet for tablet ead98cda260840c092a9e617172d2312 (DEFAULT_TABLE table=test-table [id=54d56ebc12a8485287b2d3430525a1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:49.845744 13067 tablet_service.cc:1467] Processing CreateTablet for tablet ead98cda260840c092a9e617172d2312 (DEFAULT_TABLE table=test-table [id=54d56ebc12a8485287b2d3430525a1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:49.845882 13347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead98cda260840c092a9e617172d2312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:49.845927 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead98cda260840c092a9e617172d2312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:49.847355 13207 tablet_service.cc:1467] Processing CreateTablet for tablet ead98cda260840c092a9e617172d2312 (DEFAULT_TABLE table=test-table [id=54d56ebc12a8485287b2d3430525a1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:49.847476 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead98cda260840c092a9e617172d2312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:49.848908 13443 tablet_bootstrap.cc:492] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: Bootstrap starting.
I20260923 23:05:49.849308 13443 tablet_bootstrap.cc:654] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:49.849648 13441 tablet_bootstrap.cc:492] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: Bootstrap starting.
I20260923 23:05:49.850070 13441 tablet_bootstrap.cc:654] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:49.850401 13442 tablet_bootstrap.cc:492] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: Bootstrap starting.
I20260923 23:05:49.850818 13442 tablet_bootstrap.cc:654] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:49.851639 13443 tablet_bootstrap.cc:492] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: No bootstrap required, opened a new log
I20260923 23:05:49.851691 13443 ts_tablet_manager.cc:1397] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:05:49.851848 13443 raft_consensus.cc:348] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.851920 13443 raft_consensus.cc:374] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:49.851940 13443 raft_consensus.cc:729] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb06458ffec40c897035e161d13175b, State: Initialized, Role: FOLLOWER
I20260923 23:05:49.851989 13443 consensus_queue.cc:260] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.852080 13443 ts_tablet_manager.cc:1428] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:49.852501 13442 tablet_bootstrap.cc:492] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: No bootstrap required, opened a new log
I20260923 23:05:49.852551 13442 ts_tablet_manager.cc:1397] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:49.852679 13442 raft_consensus.cc:348] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.852743 13442 raft_consensus.cc:374] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:49.852763 13442 raft_consensus.cc:729] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c89160484e40fa9ffec398b09e0aaa, State: Initialized, Role: FOLLOWER
I20260923 23:05:49.852819 13442 consensus_queue.cc:260] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.852916 13442 ts_tablet_manager.cc:1428] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:49.853386 13441 tablet_bootstrap.cc:492] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: No bootstrap required, opened a new log
I20260923 23:05:49.853440 13441 ts_tablet_manager.cc:1397] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260923 23:05:49.853574 13441 raft_consensus.cc:348] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.853639 13441 raft_consensus.cc:374] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:49.853657 13441 raft_consensus.cc:729] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f361a7f9a49f446f9f7f52ccd34052a0, State: Initialized, Role: FOLLOWER
I20260923 23:05:49.853703 13441 consensus_queue.cc:260] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.853782 13441 ts_tablet_manager.cc:1428] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:49.963830 13472 raft_consensus.cc:482] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:49.963999 13472 raft_consensus.cc:504] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.964212 13472 leader_election.cc:288] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301), f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139)
I20260923 23:05:49.964452 13366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead98cda260840c092a9e617172d2312" candidate_uuid: "38c89160484e40fa9ffec398b09e0aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" is_pre_election: true
I20260923 23:05:49.964545 13366 raft_consensus.cc:2454] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c89160484e40fa9ffec398b09e0aaa in term 0.
I20260923 23:05:49.964478 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead98cda260840c092a9e617172d2312" candidate_uuid: "38c89160484e40fa9ffec398b09e0aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb06458ffec40c897035e161d13175b" is_pre_election: true
I20260923 23:05:49.964622 13087 raft_consensus.cc:2454] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c89160484e40fa9ffec398b09e0aaa in term 0.
I20260923 23:05:49.964699 13160 leader_election.cc:302] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c89160484e40fa9ffec398b09e0aaa, f361a7f9a49f446f9f7f52ccd34052a0; no voters:
I20260923 23:05:49.964859 13472 raft_consensus.cc:2781] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:49.964923 13472 raft_consensus.cc:482] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:49.964956 13472 raft_consensus.cc:3037] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.965677 13472 raft_consensus.cc:504] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.965833 13472 leader_election.cc:288] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [CANDIDATE]: Term 1 election: Requested vote from peers 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301), f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139)
I20260923 23:05:49.966032 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead98cda260840c092a9e617172d2312" candidate_uuid: "38c89160484e40fa9ffec398b09e0aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb06458ffec40c897035e161d13175b"
I20260923 23:05:49.966128 13087 raft_consensus.cc:3037] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.966085 13366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead98cda260840c092a9e617172d2312" candidate_uuid: "38c89160484e40fa9ffec398b09e0aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f361a7f9a49f446f9f7f52ccd34052a0"
I20260923 23:05:49.966217 13366 raft_consensus.cc:3037] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:49.967003 13366 raft_consensus.cc:2454] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c89160484e40fa9ffec398b09e0aaa in term 1.
I20260923 23:05:49.967003 13087 raft_consensus.cc:2454] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c89160484e40fa9ffec398b09e0aaa in term 1.
I20260923 23:05:49.967190 13160 leader_election.cc:302] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c89160484e40fa9ffec398b09e0aaa, f361a7f9a49f446f9f7f52ccd34052a0; no voters:
I20260923 23:05:49.967301 13472 raft_consensus.cc:2781] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:49.967366 13472 raft_consensus.cc:686] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 1 LEADER]: Becoming Leader. State: Replica: 38c89160484e40fa9ffec398b09e0aaa, State: Running, Role: LEADER
I20260923 23:05:49.967422 13472 consensus_queue.cc:237] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:49.968083 12929 catalog_manager.cc:5495] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2). New cstate: current_term: 1 leader_uuid: "38c89160484e40fa9ffec398b09e0aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } health_report { overall_health: UNKNOWN } } }
I20260923 23:05:49.997609 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:49.994971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.5.74.2:37795)
0923 23:05:49.995034 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:49.995036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:49.995045 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:49.995094 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:49.995095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:49.995107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:49.995183 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:49.995304 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.995930 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.996888 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:49.997085 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:49.997162 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:49.997365 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:49.997397 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:49.997440 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:49.997526 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 23:05:49.997548 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:05:49.998588 13366 raft_consensus.cc:1261] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Refusing update from remote peer 38c89160484e40fa9ffec398b09e0aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:49.998634 13087 raft_consensus.cc:1261] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Refusing update from remote peer 38c89160484e40fa9ffec398b09e0aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:49.998878 13505 consensus_queue.cc:1035] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:49.998982 13472 consensus_queue.cc:1035] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:50.001974 12930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:50.002070 12930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:50.002784 12930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879: Sending DeleteTablet for 3 replicas of tablet ead98cda260840c092a9e617172d2312
I20260923 23:05:50.003098 13207 tablet_service.cc:1514] Processing DeleteTablet for tablet ead98cda260840c092a9e617172d2312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:49 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:38662
I20260923 23:05:50.003104 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet ead98cda260840c092a9e617172d2312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:49 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:36500
I20260923 23:05:50.003198 13511 tablet_replica.cc:330] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: stopping tablet replica
I20260923 23:05:50.003201 13512 tablet_replica.cc:330] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: stopping tablet replica
I20260923 23:05:50.003149 13347 tablet_service.cc:1514] Processing DeleteTablet for tablet ead98cda260840c092a9e617172d2312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:49 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:55186
I20260923 23:05:50.003269 13511 raft_consensus.cc:2229] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:50.003289 13513 tablet_replica.cc:330] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: stopping tablet replica
I20260923 23:05:50.003248 13512 raft_consensus.cc:2229] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:50.003334 13513 raft_consensus.cc:2229] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:50.003371 13513 raft_consensus.cc:2258] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:50.003366 13512 raft_consensus.cc:2258] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:50.003316 13511 raft_consensus.cc:2258] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:50.003333 12930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:05:50.003489 12930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:05:50.003717 13511 ts_tablet_manager.cc:1905] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:50.003904 13513 ts_tablet_manager.cc:1905] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:50.004259 13512 ts_tablet_manager.cc:1905] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:50.004715 13067 tablet_service.cc:1467] Processing CreateTablet for tablet 6ebe77a61cf74ddcadf88476fb439e85 (DEFAULT_TABLE table=test-table [id=0159cf9387614f24a19dcd603acf3435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:50.004810 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ebe77a61cf74ddcadf88476fb439e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:50.005537 13511 ts_tablet_manager.cc:1918] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:50.005540 13347 tablet_service.cc:1467] Processing CreateTablet for tablet 6ebe77a61cf74ddcadf88476fb439e85 (DEFAULT_TABLE table=test-table [id=0159cf9387614f24a19dcd603acf3435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:50.005589 13511 log.cc:1196] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/ead98cda260840c092a9e617172d2312
I20260923 23:05:50.005847 13207 tablet_service.cc:1467] Processing CreateTablet for tablet 6ebe77a61cf74ddcadf88476fb439e85 (DEFAULT_TABLE table=test-table [id=0159cf9387614f24a19dcd603acf3435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:05:50.005817 13511 ts_tablet_manager.cc:1939] T ead98cda260840c092a9e617172d2312 P 6fb06458ffec40c897035e161d13175b: Deleting consensus metadata
I20260923 23:05:50.006412 12916 catalog_manager.cc:4841] TS 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301): tablet ead98cda260840c092a9e617172d2312 (table test-table [id=54d56ebc12a8485287b2d3430525a1eb]) successfully deleted
I20260923 23:05:50.006604 13443 tablet_bootstrap.cc:492] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b: Bootstrap starting.
I20260923 23:05:50.006884 13512 ts_tablet_manager.cc:1918] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:50.006954 13443 tablet_bootstrap.cc:654] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:50.006923 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ebe77a61cf74ddcadf88476fb439e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:50.007053 13513 ts_tablet_manager.cc:1918] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:05:50.007107 13513 log.cc:1196] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/ead98cda260840c092a9e617172d2312
I20260923 23:05:50.007324 13513 ts_tablet_manager.cc:1939] T ead98cda260840c092a9e617172d2312 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting consensus metadata
I20260923 23:05:50.007485 13347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ebe77a61cf74ddcadf88476fb439e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:50.007551 13443 tablet_bootstrap.cc:492] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b: No bootstrap required, opened a new log
I20260923 23:05:50.007593 13443 ts_tablet_manager.cc:1397] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:50.007704 13443 raft_consensus.cc:348] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:50.007764 13443 raft_consensus.cc:374] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:50.007781 13443 raft_consensus.cc:729] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb06458ffec40c897035e161d13175b, State: Initialized, Role: FOLLOWER
I20260923 23:05:50.007828 13443 consensus_queue.cc:260] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:50.007905 13443 ts_tablet_manager.cc:1428] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:50.008697 13442 tablet_bootstrap.cc:492] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: Bootstrap starting.
I20260923 23:05:50.008821 13441 tablet_bootstrap.cc:492] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: Bootstrap starting.
I20260923 23:05:50.006953 13512 log.cc:1196] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/ead98cda260840c092a9e617172d2312
I20260923 23:05:50.009192 13442 tablet_bootstrap.cc:654] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:50.009223 13512 ts_tablet_manager.cc:1939] T ead98cda260840c092a9e617172d2312 P 38c89160484e40fa9ffec398b09e0aaa: Deleting consensus metadata
I20260923 23:05:50.009555 12914 catalog_manager.cc:4841] TS 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795): tablet ead98cda260840c092a9e617172d2312 (table test-table [id=54d56ebc12a8485287b2d3430525a1eb]) successfully deleted
I20260923 23:05:50.009867 13441 tablet_bootstrap.cc:654] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:50.010165 13442 tablet_bootstrap.cc:492] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: No bootstrap required, opened a new log
I20260923 23:05:50.010211 13442 ts_tablet_manager.cc:1397] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:05:50.010352 12916 catalog_manager.cc:4841] TS f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139): tablet ead98cda260840c092a9e617172d2312 (table test-table [id=54d56ebc12a8485287b2d3430525a1eb]) successfully deleted
I20260923 23:05:50.010353 13442 raft_consensus.cc:348] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:50.010428 13442 raft_consensus.cc:374] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:50.010454 13442 raft_consensus.cc:729] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c89160484e40fa9ffec398b09e0aaa, State: Initialized, Role: FOLLOWER
I20260923 23:05:50.010510 13442 consensus_queue.cc:260] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:50.010610 13442 ts_tablet_manager.cc:1428] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:50.010774 13441 tablet_bootstrap.cc:492] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: No bootstrap required, opened a new log
I20260923 23:05:50.010859 13441 ts_tablet_manager.cc:1397] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:05:50.011004 13441 raft_consensus.cc:348] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:50.011106 13441 raft_consensus.cc:374] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:50.011139 13441 raft_consensus.cc:729] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f361a7f9a49f446f9f7f52ccd34052a0, State: Initialized, Role: FOLLOWER
I20260923 23:05:50.011196 13441 consensus_queue.cc:260] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:50.011289 13441 ts_tablet_manager.cc:1428] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:50.019333 13447 raft_consensus.cc:482] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:05:50.019449 13447 raft_consensus.cc:504] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:50.019815 13447 leader_election.cc:288] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301), 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795)
I20260923 23:05:50.023089 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.020142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:53455 (local address 127.5.74.1:39301)
0923 23:05:50.020234 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:50.020236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:50.020296 (+ 60us) server_negotiation.cc:407] Connection header received
0923 23:05:50.020385 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:50.020386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:50.020395 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:50.020471 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:50.020688 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.021347 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.022358 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.022744 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.022804 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:50.022813 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:50.022970 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 23:05:50.023015 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 23:05:50.023100 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.020075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:39301 (local address 127.5.74.3:53455)
0923 23:05:50.020231 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:50.020241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:50.020312 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:50.020483 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:50.020483 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:50.020485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:50.020565 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:50.020567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.021434 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.021435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:50.022267 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:50.022269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.022826 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.022827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:50.022895 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:50.022899 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:05:50.022953 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 23:05:50.022990 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":29,"thread_start_us":97,"threads_started":1}
I20260923 23:05:50.023166 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.020101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:40139 (local address 127.5.74.2:37795)
0923 23:05:50.020175 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:50.020178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:50.020288 (+ 110us) server_negotiation.cc:407] Connection header received
0923 23:05:50.020357 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:50.020358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:50.020367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:50.020425 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:05:50.020613 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.021281 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.022405 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.022763 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.022863 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:50.022873 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:50.023052 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 23:05:50.023090 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:05:50.023227 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.019980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:37795 (local address 127.5.74.3:40139)
0923 23:05:50.020193 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:50.020206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:50.020282 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:50.020443 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:50.020444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:50.020446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:05:50.020534 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:50.020537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.021378 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.021379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:50.022283 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:50.022285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.022872 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.022873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:50.022970 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:50.022975 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:05:50.023039 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:05:50.023081 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":50,"thread_start_us":61,"threads_started":1}
I20260923 23:05:50.023587 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ebe77a61cf74ddcadf88476fb439e85" candidate_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c89160484e40fa9ffec398b09e0aaa" is_pre_election: true
I20260923 23:05:50.023605 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ebe77a61cf74ddcadf88476fb439e85" candidate_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb06458ffec40c897035e161d13175b" is_pre_election: true
I20260923 23:05:50.023706 13227 raft_consensus.cc:2454] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f361a7f9a49f446f9f7f52ccd34052a0 in term 0.
I20260923 23:05:50.023701 13087 raft_consensus.cc:2454] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f361a7f9a49f446f9f7f52ccd34052a0 in term 0.
I20260923 23:05:50.023892 13298 leader_election.cc:302] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c89160484e40fa9ffec398b09e0aaa, f361a7f9a49f446f9f7f52ccd34052a0; no voters:
I20260923 23:05:50.024124 13447 raft_consensus.cc:2781] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:05:50.024214 13447 raft_consensus.cc:482] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:05:50.024240 13447 raft_consensus.cc:3037] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:50.025504 13447 raft_consensus.cc:504] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:50.025673 13447 leader_election.cc:288] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [CANDIDATE]: Term 1 election: Requested vote from peers 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301), 38c89160484e40fa9ffec398b09e0aaa (127.5.74.2:37795)
I20260923 23:05:50.025835 13087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ebe77a61cf74ddcadf88476fb439e85" candidate_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb06458ffec40c897035e161d13175b"
I20260923 23:05:50.025913 13087 raft_consensus.cc:3037] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:50.025872 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ebe77a61cf74ddcadf88476fb439e85" candidate_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c89160484e40fa9ffec398b09e0aaa"
I20260923 23:05:50.025962 13227 raft_consensus.cc:3037] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:50.026556 13087 raft_consensus.cc:2454] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f361a7f9a49f446f9f7f52ccd34052a0 in term 1.
I20260923 23:05:50.026643 13227 raft_consensus.cc:2454] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f361a7f9a49f446f9f7f52ccd34052a0 in term 1.
I20260923 23:05:50.026746 13300 leader_election.cc:302] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb06458ffec40c897035e161d13175b, f361a7f9a49f446f9f7f52ccd34052a0; no voters:
I20260923 23:05:50.026862 13447 raft_consensus.cc:2781] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:50.027029 13447 raft_consensus.cc:686] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 LEADER]: Becoming Leader. State: Replica: f361a7f9a49f446f9f7f52ccd34052a0, State: Running, Role: LEADER
I20260923 23:05:50.027122 13447 consensus_queue.cc:237] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } }
I20260923 23:05:50.027897 12929 catalog_manager.cc:5495] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3). New cstate: current_term: 1 leader_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f361a7f9a49f446f9f7f52ccd34052a0" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35139 } health_report { overall_health: HEALTHY } } }
I20260923 23:05:50.035487 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.032727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.5.74.3:35139)
0923 23:05:50.032901 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:50.032903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:50.032916 (+ 13us) server_negotiation.cc:407] Connection header received
0923 23:05:50.032953 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:50.032954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:50.032971 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:50.033072 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:50.033183 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.033746 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.034562 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.034860 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.034947 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:50.035112 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:50.035234 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:50.035321 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:50.035366 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 23:05:50.035385 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260923 23:05:50.040656 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.038001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:39301 (local address 127.5.74.3:54901)
0923 23:05:50.038181 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:50.038192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:50.038254 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:50.038374 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:50.038375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:50.038377 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:50.038426 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:50.038427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.039004 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.039004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:50.039603 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:50.039605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.040083 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.040084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:50.040146 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:50.040199 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:50.040446 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:50.040448 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:05:50.040518 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 23:05:50.040565 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":114,"threads_started":1}
I20260923 23:05:50.040645 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.038096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:54901 (local address 127.5.74.1:39301)
0923 23:05:50.038144 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:50.038146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:50.038240 (+ 94us) server_negotiation.cc:407] Connection header received
0923 23:05:50.038294 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:50.038295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:50.038303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:50.038360 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:50.038469 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.038934 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.039714 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.039990 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.040054 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:50.040274 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:50.040308 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:50.040442 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:50.040527 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 23:05:50.040552 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:05:50.043527 13227 raft_consensus.cc:1261] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Refusing update from remote peer f361a7f9a49f446f9f7f52ccd34052a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:50.043603 13087 raft_consensus.cc:1261] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Refusing update from remote peer f361a7f9a49f446f9f7f52ccd34052a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:05:50.043802 13538 consensus_queue.cc:1035] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c89160484e40fa9ffec398b09e0aaa" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:50.043926 13447 consensus_queue.cc:1035] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb06458ffec40c897035e161d13175b" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 39301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:05:50.051033 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.048546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:52471 (local address 127.5.74.3:35139)
0923 23:05:50.048589 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:50.048591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:50.048664 (+ 73us) server_negotiation.cc:407] Connection header received
0923 23:05:50.048759 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:50.048760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:50.048770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:50.048819 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:05:50.048926 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.049418 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.050203 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.050489 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.050556 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:50.050768 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:05:50.050801 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:05:50.050855 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:50.050932 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 23:05:50.050955 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:05:50.051035 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.048473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:35139 (local address 127.5.74.1:52471)
0923 23:05:50.048589 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:50.048607 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:50.048672 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:50.048821 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:50.048822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:50.048823 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:05:50.048869 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:50.048870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.049504 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.049505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:50.050117 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:50.050120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:50.050611 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:50.050612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:50.050680 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:50.050689 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:05:50.050868 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:05:50.050871 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:05:50.050912 (+ 41us) client_negotiation.cc:240] Negotiation successful
0923 23:05:50.050968 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 23:05:50.057862 12930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:50.058010 12930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:05:50.058748 12930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83925fa9376848f3a1bbc2b65be60879: Sending DeleteTablet for 3 replicas of tablet 6ebe77a61cf74ddcadf88476fb439e85
I20260923 23:05:50.059053 13347 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ebe77a61cf74ddcadf88476fb439e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:50 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:55186
I20260923 23:05:50.059062 13207 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ebe77a61cf74ddcadf88476fb439e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:50 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:38662
I20260923 23:05:50.059151 13513 tablet_replica.cc:330] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: stopping tablet replica
I20260923 23:05:50.059182 13512 tablet_replica.cc:330] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: stopping tablet replica
I20260923 23:05:50.059206 13513 raft_consensus.cc:2229] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:05:50.059259 13512 raft_consensus.cc:2229] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:50.059304 13512 raft_consensus.cc:2258] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:50.059319 13513 raft_consensus.cc:2258] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:50.059358 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ebe77a61cf74ddcadf88476fb439e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:05:50 UTC) from {username='oryx', principal='oryx/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:36500
I20260923 23:05:50.059549 13511 tablet_replica.cc:330] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b: stopping tablet replica
I20260923 23:05:50.059610 13511 raft_consensus.cc:2229] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:05:50.059648 13511 raft_consensus.cc:2258] T 6ebe77a61cf74ddcadf88476fb439e85 P 6fb06458ffec40c897035e161d13175b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:05:50.059762 13512 ts_tablet_manager.cc:1905] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:50.060046 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 13002
I20260923 23:05:50.061089 13513 ts_tablet_manager.cc:1905] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:05:50.063354 13513 ts_tablet_manager.cc:1918] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:05:50.063508 13513 log.cc:1196] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/6ebe77a61cf74ddcadf88476fb439e85
I20260923 23:05:50.063721 13513 ts_tablet_manager.cc:1939] T 6ebe77a61cf74ddcadf88476fb439e85 P f361a7f9a49f446f9f7f52ccd34052a0: Deleting consensus metadata
I20260923 23:05:50.064069 12916 catalog_manager.cc:4841] TS f361a7f9a49f446f9f7f52ccd34052a0 (127.5.74.3:35139): tablet 6ebe77a61cf74ddcadf88476fb439e85 (table test-table [id=0159cf9387614f24a19dcd603acf3435]) successfully deleted
I20260923 23:05:50.069640 13512 ts_tablet_manager.cc:1918] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
W20260923 23:05:50.069618 12916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:39301 (error 108)
I20260923 23:05:50.069725 13512 log.cc:1196] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/6ebe77a61cf74ddcadf88476fb439e85
I20260923 23:05:50.069939 13512 ts_tablet_manager.cc:1939] T 6ebe77a61cf74ddcadf88476fb439e85 P 38c89160484e40fa9ffec398b09e0aaa: Deleting consensus metadata
I20260923 23:05:50.070302 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 13142
I20260923 23:05:50.076813 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:50.076533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:39301 (local address 127.0.0.1:50770)
0923 23:05:50.076694 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:50.076751 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:39301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
W20260923 23:05:50.077132 13552 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:50.076794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:37795 (local address 127.0.0.1:54740)
0923 23:05:50.076949 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:50.076962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:50.077039 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:50.077082 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.2:37795: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
W20260923 23:05:50.077188 12916 catalog_manager.cc:4584] TS 6fb06458ffec40c897035e161d13175b (127.5.74.1:39301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ebe77a61cf74ddcadf88476fb439e85: Network error: Client connection negotiation failed: client connection to 127.5.74.1:39301: connect: Connection refused (error 111)
I20260923 23:05:50.077345 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 13282
I20260923 23:05:50.083355 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 12898
2026-09-23T23:05:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3359 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:50 dist-test-slave-j4c5 krb5kdc[13555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:50 dist-test-slave-j4c5 krb5kdc[13555](info): set up 2 sockets
Sep 23 23:05:50 dist-test-slave-j4c5 krb5kdc[13555](info): commencing operation
krb5kdc: starting...
W20260923 23:05:51.173398 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:51 dist-test-slave-j4c5 krb5kdc[13555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:51.312294 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:42123
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:51.399631 13568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:51.399883 13568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:51.399919 13568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:51.401230 13568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:51.401302 13568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:51.401333 13568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:51.401366 13568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:51.401393 13568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:51.402969 13568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:42123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:51.403255 13568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:51.403493 13568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:51.408478 13576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:51.408619 13573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:51.408671 13568 server_base.cc:1034] running on GCE node
W20260923 23:05:51.408466 13574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:51.408996 13568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:51.409250 13568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:51.410436 13568 hybrid_clock.cc:648] HybridClock initialized: now 1790204751410428 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T23:05:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1334 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:05:51 dist-test-slave-j4c5 krb5kdc[13581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:05:51 dist-test-slave-j4c5 krb5kdc[13581](info): set up 2 sockets
Sep 23 23:05:51 dist-test-slave-j4c5 krb5kdc[13581](info): commencing operation
krb5kdc: starting...
W20260923 23:05:52.492945 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:05:52 dist-test-slave-j4c5 krb5kdc[13581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:05:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:05:52.628574 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:39281
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:39281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:52.712458 13594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:52.712682 13594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:52.712706 13594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:52.713922 13594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:52.713977 13594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:52.713992 13594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:52.714005 13594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:52.714017 13594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:52.715538 13594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:39281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:39281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:52.715803 13594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:52.716043 13594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:52.720969 13599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:52.721004 13600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:52.721156 13602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:52.721315 13594 server_base.cc:1034] running on GCE node
I20260923 23:05:52.721575 13594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:52.721886 13594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:52.723083 13594 hybrid_clock.cc:648] HybridClock initialized: now 1790204752723063 us; error 32 us; skew 500 ppm
Sep 23 23:05:52 dist-test-slave-j4c5 krb5kdc[13581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204752, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:52.725768 13594 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:52.726248 13594 webserver.cc:460] Webserver started at http://127.5.74.62:38991/ using document root <none> and password file <none>
I20260923 23:05:52.726487 13594 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:52.726554 13594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:52.726702 13594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:52.727849 13594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "8877cae48e334ff6b993289ab0d3bfa5"
format_stamp: "Formatted at 2026-09-23 23:05:52 on dist-test-slave-j4c5"
server_key: "b9fe963deb5c38c712db89db80477349"
server_key_iv: "ca5a76fae8fd643e7761eded5b43e7ed"
server_key_version: "encryptionkey@0"
I20260923 23:05:52.728281 13594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "8877cae48e334ff6b993289ab0d3bfa5"
format_stamp: "Formatted at 2026-09-23 23:05:52 on dist-test-slave-j4c5"
server_key: "b9fe963deb5c38c712db89db80477349"
server_key_iv: "ca5a76fae8fd643e7761eded5b43e7ed"
server_key_version: "encryptionkey@0"
I20260923 23:05:52.729683 13594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:52.730530 13609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:52.730697 13594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:05:52.730798 13594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "8877cae48e334ff6b993289ab0d3bfa5"
format_stamp: "Formatted at 2026-09-23 23:05:52 on dist-test-slave-j4c5"
server_key: "b9fe963deb5c38c712db89db80477349"
server_key_iv: "ca5a76fae8fd643e7761eded5b43e7ed"
server_key_version: "encryptionkey@0"
I20260923 23:05:52.730880 13594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:52.739235 13594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:52.741235 13594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:52.741387 13594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:52.745368 13594 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:39281
I20260923 23:05:52.745476 13661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:39281 every 8 connection(s)
I20260923 23:05:52.745807 13594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:52.746757 13662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:05:52.749338 13662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: Bootstrap starting.
I20260923 23:05:52.750202 13662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: Neither blocks nor log segments found. Creating new log.
I20260923 23:05:52.750988 13662 log.cc:824] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:52.752108 13662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: No bootstrap required, opened a new log
I20260923 23:05:52.753803 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 13594
I20260923 23:05:52.753890 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:05:52.754000 13662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } }
I20260923 23:05:52.754175 13662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:05:52.754215 5416 external_mini_cluster.cc:1428] Setting key 93d4bc17c17612ed38f1a3f1aa6d5963
I20260923 23:05:52.754223 13662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8877cae48e334ff6b993289ab0d3bfa5, State: Initialized, Role: FOLLOWER
I20260923 23:05:52.754406 13662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } }
I20260923 23:05:52.754503 13662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:52.754542 13662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:52.754627 13662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:05:52.755357 13662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } }
I20260923 23:05:52.755482 13662 leader_election.cc:302] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8877cae48e334ff6b993289ab0d3bfa5; no voters:
I20260923 23:05:52.755697 13662 leader_election.cc:288] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:05:52.755734 13667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:05:52.755849 13667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 1 LEADER]: Becoming Leader. State: Replica: 8877cae48e334ff6b993289ab0d3bfa5, State: Running, Role: LEADER
I20260923 23:05:52.755991 13662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:52.756009 13667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } }
I20260923 23:05:52.756563 13668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8877cae48e334ff6b993289ab0d3bfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } } }
I20260923 23:05:52.756829 13668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:52.756762 13669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8877cae48e334ff6b993289ab0d3bfa5. Latest consensus state: current_term: 1 leader_uuid: "8877cae48e334ff6b993289ab0d3bfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } } }
I20260923 23:05:52.757237 13669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: This master's current role is: LEADER
W20260923 23:05:52.757936 13683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:05:52.757997 13683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:05:52.758654 13676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:52.758888 13676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:52.760560 13676 catalog_manager.cc:1347] Generated new cluster ID: 95c660e3c3844efdbbfa4ac26fe0a874
I20260923 23:05:52.760617 13676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:05:52 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:52.766250 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:52.754720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.5.74.62:39281)
0923 23:05:52.754927 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:52.754937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:52.754974 (+ 37us) server_negotiation.cc:407] Connection header received
0923 23:05:52.755306 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:52.755312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:52.755358 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:52.755520 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:52.756335 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:52.757784 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:52.758647 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:52.759257 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:52.759374 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:52.762431 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:52.762441 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:52.762442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:52.762458 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:52.764482 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:52.764911 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:52.764914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:52.764914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:52.765042 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:52.765268 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:52.765269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:52.765269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:52.765589 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:52.765742 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:52.765874 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 23:05:52.766038 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":122,"threads_started":1}
I20260923 23:05:52.769210 13676 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:05:52.769933 13676 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:52.783514 13676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: Generated new TSK 0
I20260923 23:05:52.783864 13676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:05:52.815655 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39281
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:52.923482 13688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:52.923703 13688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:52.923728 13688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:52.925067 13688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:52.925122 13688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:52.925150 13688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:52.926789 13688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:52.927068 13688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:52.927307 13688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:52.932412 13694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:52.932448 13696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:52.932396 13693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:52.932696 13688 server_base.cc:1034] running on GCE node
I20260923 23:05:52.932979 13688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:52.933202 13688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:52.934346 13688 hybrid_clock.cc:648] HybridClock initialized: now 1790204752934332 us; error 32 us; skew 500 ppm
Sep 23 23:05:52 dist-test-slave-j4c5 krb5kdc[13581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204752, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:52.937033 13688 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:05:52.937498 13688 webserver.cc:460] Webserver started at http://127.5.74.1:41427/ using document root <none> and password file <none>
I20260923 23:05:52.937738 13688 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:52.937803 13688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:52.937954 13688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:52.939029 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "e29cf300b9bc484db767a0a2fbee99bf"
format_stamp: "Formatted at 2026-09-23 23:05:52 on dist-test-slave-j4c5"
server_key: "fd57f13e558be9fc9cb6a3cf62c45685"
server_key_iv: "508c10ff390461283f5c347e279972ca"
server_key_version: "encryptionkey@0"
I20260923 23:05:52.939393 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "e29cf300b9bc484db767a0a2fbee99bf"
format_stamp: "Formatted at 2026-09-23 23:05:52 on dist-test-slave-j4c5"
server_key: "fd57f13e558be9fc9cb6a3cf62c45685"
server_key_iv: "508c10ff390461283f5c347e279972ca"
server_key_version: "encryptionkey@0"
I20260923 23:05:52.940873 13688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 23:05:52.941766 13703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:52.942013 13688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:52.942111 13688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "e29cf300b9bc484db767a0a2fbee99bf"
format_stamp: "Formatted at 2026-09-23 23:05:52 on dist-test-slave-j4c5"
server_key: "fd57f13e558be9fc9cb6a3cf62c45685"
server_key_iv: "508c10ff390461283f5c347e279972ca"
server_key_version: "encryptionkey@0"
I20260923 23:05:52.942183 13688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:52.956408 13688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:52.959328 13688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:52.959515 13688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:52.959771 13688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:52.960109 13688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:52.960180 13688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:52.960222 13688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:52.960242 13688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:52.966380 13688 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:41989
I20260923 23:05:52.966464 13816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:41989 every 8 connection(s)
I20260923 23:05:52.966969 13688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:52.971930 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 13688
I20260923 23:05:52.972082 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:05:52.972352 5416 external_mini_cluster.cc:1428] Setting key d77ddb147fa1c3d6b69c89e548ee7caf
Sep 23 23:05:52 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204752, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:52.978037 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:52.967848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:32991 (local address 127.5.74.62:39281)
0923 23:05:52.967914 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:52.967917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:52.968087 (+ 170us) server_negotiation.cc:407] Connection header received
0923 23:05:52.968682 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:52.968684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:52.968700 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:52.968755 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:52.969744 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:52.970432 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:52.971525 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:52.971738 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:52.971835 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:52.974702 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:52.974704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:52.974704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:52.974718 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:52.976376 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:52.976910 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:52.976912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:52.976912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:52.976977 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:52.977315 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:52.977316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:52.977316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:52.977436 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:52.977521 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:52.977835 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 23:05:52.977931 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:05:52.978287 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:52.967796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:32991)
0923 23:05:52.967954 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:52.967998 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:52.968570 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:52.968831 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:52.968835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:52.969437 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:52.969636 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:52.969640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:52.970562 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:52.970564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:52.971423 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:52.971426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:52.971850 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:52.971851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:52.971988 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:52.972866 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:52.972877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:52.974576 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:52.976486 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:52.976496 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:52.976499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:52.976755 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:52.977048 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:52.977049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:52.977050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:52.977180 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:52.977511 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:52.977518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:52.977673 (+ 155us) client_negotiation.cc:769] Sending connection context
0923 23:05:52.977809 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 23:05:52.977997 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260923 23:05:52.978848 13817 heartbeater.cc:347] Connected to a master server at 127.5.74.62:39281
I20260923 23:05:52.978971 13817 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:52.979173 13817 heartbeater.cc:511] Master 127.5.74.62:39281 requested a full tablet report, sending...
I20260923 23:05:52.979878 13626 ts_manager.cc:194] Registered new tserver with Master: e29cf300b9bc484db767a0a2fbee99bf (127.5.74.1:41989)
I20260923 23:05:52.980652 13626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:32991
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:05:52.999796 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39281
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:53.086627 13822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:53.086859 13822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:53.086886 13822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:53.088371 13822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:53.088477 13822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:53.088513 13822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:53.090420 13822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:53.090724 13822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:53.090984 13822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:53.096832 13828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:53.096832 13830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:53.096961 13827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:53.097107 13822 server_base.cc:1034] running on GCE node
I20260923 23:05:53.097422 13822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:53.097648 13822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:53.098819 13822 hybrid_clock.cc:648] HybridClock initialized: now 1790204753098801 us; error 38 us; skew 500 ppm
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204753, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:53.101823 13822 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:05:53.102417 13822 webserver.cc:460] Webserver started at http://127.5.74.2:33127/ using document root <none> and password file <none>
I20260923 23:05:53.102635 13822 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:53.102689 13822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:53.102790 13822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:53.103997 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "87b00849b73c4060b413b191b3ea0ca7"
format_stamp: "Formatted at 2026-09-23 23:05:53 on dist-test-slave-j4c5"
server_key: "0373c3cc038be42ac538ad3fb302a784"
server_key_iv: "2d0a472b51e256bb58d011e06e2def87"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.104499 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "87b00849b73c4060b413b191b3ea0ca7"
format_stamp: "Formatted at 2026-09-23 23:05:53 on dist-test-slave-j4c5"
server_key: "0373c3cc038be42ac538ad3fb302a784"
server_key_iv: "2d0a472b51e256bb58d011e06e2def87"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.106242 13822 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 23:05:53.107131 13837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.107301 13822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:53.107407 13822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "87b00849b73c4060b413b191b3ea0ca7"
format_stamp: "Formatted at 2026-09-23 23:05:53 on dist-test-slave-j4c5"
server_key: "0373c3cc038be42ac538ad3fb302a784"
server_key_iv: "2d0a472b51e256bb58d011e06e2def87"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.107524 13822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:53.117376 13822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:53.120277 13822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:53.120473 13822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:53.120743 13822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:53.121170 13822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:53.121217 13822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.121261 13822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:53.121282 13822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.129500 13822 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:41671
I20260923 23:05:53.129570 13950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:41671 every 8 connection(s)
I20260923 23:05:53.129953 13822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:05:53.135560 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 13822
I20260923 23:05:53.135667 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:05:53.135814 5416 external_mini_cluster.cc:1428] Setting key 2959e9e629a1ce00ef12871599288dae
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204753, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
I20260923 23:05:53.145171 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.130782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:55879 (local address 127.5.74.62:39281)
0923 23:05:53.130835 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:53.130837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:53.131095 (+ 258us) server_negotiation.cc:407] Connection header received
0923 23:05:53.131632 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:53.131633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:53.131647 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:53.131694 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:53.132755 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.133256 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.134269 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.134511 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.134577 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.141937 (+ 7360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:53.141939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:53.141940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:53.141948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:53.143461 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:53.143926 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:53.143928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:53.143929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:53.143990 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:53.144399 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:53.144400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:53.144401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:53.144580 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:53.144657 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:53.144963 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 23:05:53.145063 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:05:53.145462 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.130736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:55879)
0923 23:05:53.130946 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:53.130998 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:53.131521 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:53.131773 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:53.131777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:53.132445 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:53.132653 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.132658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.133371 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.133372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.134176 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.134178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.134583 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.134585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.134706 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.139642 (+ 4936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:53.139655 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:53.141783 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:53.143577 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:53.143587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:53.143590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:53.143847 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:53.144056 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:53.144057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:53.144058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:53.144276 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:53.144647 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:53.144654 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:53.144814 (+ 160us) client_negotiation.cc:769] Sending connection context
0923 23:05:53.144932 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 23:05:53.145159 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
I20260923 23:05:53.146004 13951 heartbeater.cc:347] Connected to a master server at 127.5.74.62:39281
I20260923 23:05:53.146132 13951 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:53.146359 13951 heartbeater.cc:511] Master 127.5.74.62:39281 requested a full tablet report, sending...
I20260923 23:05:53.146687 13626 ts_manager.cc:194] Registered new tserver with Master: 87b00849b73c4060b413b191b3ea0ca7 (127.5.74.2:41671)
I20260923 23:05:53.147126 13626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:55879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:05:53.167913 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39281
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:53.253157 13956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:53.253386 13956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:53.253414 13956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:53.254664 13956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:53.254712 13956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:53.254741 13956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:53.256503 13956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:53.256768 13956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:53.257017 13956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:53.262173 13964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:53.262161 13962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:53.262166 13961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:53.262306 13956 server_base.cc:1034] running on GCE node
I20260923 23:05:53.262621 13956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:53.262874 13956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:53.264048 13956 hybrid_clock.cc:648] HybridClock initialized: now 1790204753264045 us; error 47 us; skew 500 ppm
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204753, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:53.266710 13956 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:05:53.267184 13956 webserver.cc:460] Webserver started at http://127.5.74.3:44075/ using document root <none> and password file <none>
I20260923 23:05:53.267462 13956 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:53.267511 13956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:53.267635 13956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:05:53.268761 13956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "4ebe546a7ea2428ea111521d53f5c2f1"
format_stamp: "Formatted at 2026-09-23 23:05:53 on dist-test-slave-j4c5"
server_key: "af9a0e18f8e3e8e24aef9066d3c264d6"
server_key_iv: "2ad68568fac3ff6fca4bba08b16dca07"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.269105 13956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "4ebe546a7ea2428ea111521d53f5c2f1"
format_stamp: "Formatted at 2026-09-23 23:05:53 on dist-test-slave-j4c5"
server_key: "af9a0e18f8e3e8e24aef9066d3c264d6"
server_key_iv: "2ad68568fac3ff6fca4bba08b16dca07"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.270478 13956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:53.271361 13971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.271616 13956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:53.271713 13956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "4ebe546a7ea2428ea111521d53f5c2f1"
format_stamp: "Formatted at 2026-09-23 23:05:53 on dist-test-slave-j4c5"
server_key: "af9a0e18f8e3e8e24aef9066d3c264d6"
server_key_iv: "2ad68568fac3ff6fca4bba08b16dca07"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.271788 13956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:53.309298 13956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:53.311717 13956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:53.311915 13956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:53.312214 13956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:53.312559 13956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:53.312595 13956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.312618 13956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:53.312631 13956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.319201 13956 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:36551
I20260923 23:05:53.319235 14084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:36551 every 8 connection(s)
I20260923 23:05:53.319615 13956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:05:53.324118 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 13956
I20260923 23:05:53.324270 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:05:53.324483 5416 external_mini_cluster.cc:1428] Setting key 85b02432d2c9c2c860c5ba4cf9e84efc
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204753, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:05:53.329118 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.320446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:54199 (local address 127.5.74.62:39281)
0923 23:05:53.320536 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:53.320538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:53.320650 (+ 112us) server_negotiation.cc:407] Connection header received
0923 23:05:53.321114 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:53.321115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:53.321125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:53.321197 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:53.322098 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.322595 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.323633 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.323831 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.323889 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.326323 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:53.326325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:53.326325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:53.326331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:53.327750 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:53.328133 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:53.328134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:53.328135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:53.328233 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:53.328543 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:53.328544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:53.328544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:53.328654 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:53.328708 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:53.328988 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 23:05:53.329052 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 23:05:53.329408 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.320363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:54199)
0923 23:05:53.320521 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:53.320564 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:53.321030 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:53.321251 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:53.321255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:53.321814 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:53.322009 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.322013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.322686 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.322687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.323538 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.323541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.323911 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.323912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.324017 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.324685 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:53.324696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:53.326235 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:05:53.327836 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:53.327843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:53.327846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:53.328069 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:53.328372 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:05:53.328372 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:05:53.328373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:05:53.328477 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:05:53.328709 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:05:53.328713 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:05:53.328836 (+ 123us) client_negotiation.cc:769] Sending connection context
0923 23:05:53.328927 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 23:05:53.329124 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260923 23:05:53.329948 14085 heartbeater.cc:347] Connected to a master server at 127.5.74.62:39281
I20260923 23:05:53.330066 14085 heartbeater.cc:464] Registering TS with master...
I20260923 23:05:53.330264 14085 heartbeater.cc:511] Master 127.5.74.62:39281 requested a full tablet report, sending...
I20260923 23:05:53.330672 13626 ts_manager.cc:194] Registered new tserver with Master: 4ebe546a7ea2428ea111521d53f5c2f1 (127.5.74.3:36551)
I20260923 23:05:53.331080 13626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:54199
I20260923 23:05:53.337002 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab.
Entry for principal oryx/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab.
WARNING: no policy specified for oryx/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab.
Entry for principal oryx/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab.
WARNING: no policy specified for oryx/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab.
Entry for principal oryx/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab.
I20260923 23:05:53.421094 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 13688
I20260923 23:05:53.425786 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 13822
I20260923 23:05:53.430116 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 13956
I20260923 23:05:53.434468 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 13594
I20260923 23:05:53.439594 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:39281
--webserver_interface=127.5.74.62
--webserver_port=38991
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:39281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:05:53.521821 14094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:53.522071 14094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:53.522096 14094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:53.523361 14094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:05:53.523414 14094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:53.523429 14094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:53.523442 14094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:05:53.523454 14094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:05:53.525063 14094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:39281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:39281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=38991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:53.525344 14094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:53.525601 14094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:53.531119 14099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:53.531167 14102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:53.531131 14100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:53.531388 14094 server_base.cc:1034] running on GCE node
I20260923 23:05:53.531679 14094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:53.532053 14094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:53.533239 14094 hybrid_clock.cc:648] HybridClock initialized: now 1790204753533212 us; error 63 us; skew 500 ppm
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204753, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:53.536121 14094 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:05:53.536705 14094 webserver.cc:460] Webserver started at http://127.5.74.62:38991/ using document root <none> and password file <none>
I20260923 23:05:53.536984 14094 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:53.537071 14094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:53.538657 14094 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:53.539502 14109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.539718 14094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:53.539832 14094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "8877cae48e334ff6b993289ab0d3bfa5"
format_stamp: "Formatted at 2026-09-23 23:05:52 on dist-test-slave-j4c5"
server_key: "b9fe963deb5c38c712db89db80477349"
server_key_iv: "ca5a76fae8fd643e7761eded5b43e7ed"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.540271 14094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:53.556383 14094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:53.558678 14094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:53.558863 14094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:53.563072 14094 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:39281
I20260923 23:05:53.563120 14161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:39281 every 8 connection(s)
I20260923 23:05:53.563529 14094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:05:53.564680 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 14094
I20260923 23:05:53.564981 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:41989
--local_ip_for_outbound_sockets=127.5.74.1
--tserver_master_addrs=127.5.74.62:39281
--webserver_port=41427
--webserver_interface=127.5.74.1
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.74.1
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:05:53.565485 14162 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:05:53.566618 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: Bootstrap starting.
I20260923 23:05:53.576884 14162 log.cc:824] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: Log is configured to *not* fsync() on all Append() calls
I20260923 23:05:53.579257 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:05:53.579589 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: Bootstrap complete.
I20260923 23:05:53.582824 14162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } }
I20260923 23:05:53.583173 14162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8877cae48e334ff6b993289ab0d3bfa5, State: Initialized, Role: FOLLOWER
I20260923 23:05:53.583338 14162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } }
I20260923 23:05:53.583405 14162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:05:53.583452 14162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:05:53.583498 14162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:05:53.584935 14162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } }
I20260923 23:05:53.585072 14162 leader_election.cc:302] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8877cae48e334ff6b993289ab0d3bfa5; no voters:
I20260923 23:05:53.585292 14162 leader_election.cc:288] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 23:05:53.585363 14166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:05:53.585544 14166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [term 2 LEADER]: Becoming Leader. State: Replica: 8877cae48e334ff6b993289ab0d3bfa5, State: Running, Role: LEADER
I20260923 23:05:53.585588 14162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:05:53.585656 14166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } }
I20260923 23:05:53.585944 14166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8877cae48e334ff6b993289ab0d3bfa5. Latest consensus state: current_term: 2 leader_uuid: "8877cae48e334ff6b993289ab0d3bfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } } }
I20260923 23:05:53.586010 14166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:53.586194 14172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:05:53.586148 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8877cae48e334ff6b993289ab0d3bfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8877cae48e334ff6b993289ab0d3bfa5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39281 } } }
I20260923 23:05:53.586364 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5 [sys.catalog]: This master's current role is: LEADER
I20260923 23:05:53.586656 14172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:05:53.586787 14172 catalog_manager.cc:1259] Loaded cluster ID: 95c660e3c3844efdbbfa4ac26fe0a874
I20260923 23:05:53.586822 14172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:05:53.588070 14172 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:05:53.588307 14172 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8877cae48e334ff6b993289ab0d3bfa5: Loaded TSK: 0
I20260923 23:05:53.588543 14172 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:05:53.670650 14164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:53.670893 14164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:53.670943 14164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:53.672122 14164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:53.672205 14164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:53.672267 14164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:05:53.673981 14164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.1.keytab
--principal=oryx/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:41989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=41427
--tserver_master_addrs=127.5.74.62:39281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:53.674232 14164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:53.674496 14164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:53.679548 14188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:53.679522 14189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:53.679797 14164 server_base.cc:1034] running on GCE node
W20260923 23:05:53.680008 14191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:53.680367 14164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:53.680678 14164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:53.681860 14164 hybrid_clock.cc:648] HybridClock initialized: now 1790204753681839 us; error 39 us; skew 500 ppm
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204753, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:53.684701 14164 init.cc:377] Logged in from keytab as oryx/127.5.74.1@KRBTEST.COM (short username oryx)
I20260923 23:05:53.685283 14164 webserver.cc:460] Webserver started at http://127.5.74.1:41427/ using document root <none> and password file <none>
I20260923 23:05:53.685554 14164 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:53.685647 14164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:53.687485 14164 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:53.688244 14198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.688446 14164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:53.688544 14164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "e29cf300b9bc484db767a0a2fbee99bf"
format_stamp: "Formatted at 2026-09-23 23:05:52 on dist-test-slave-j4c5"
server_key: "fd57f13e558be9fc9cb6a3cf62c45685"
server_key_iv: "508c10ff390461283f5c347e279972ca"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.689002 14164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:53.703884 14164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:53.706754 14164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:53.706964 14164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:53.707229 14164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:53.707576 14164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:53.707620 14164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.707652 14164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:53.707674 14164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.715144 14164 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:41989
I20260923 23:05:53.715183 14311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:41989 every 8 connection(s)
I20260923 23:05:53.715596 14164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:05:53.722455 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 14164
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:53.722781 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:41671
--local_ip_for_outbound_sockets=127.5.74.2
--tserver_master_addrs=127.5.74.62:39281
--webserver_port=33127
--webserver_interface=127.5.74.2
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.74.2
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:53.724687 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.716384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:47947)
0923 23:05:53.716543 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:53.716585 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:53.717049 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:53.717569 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:53.717573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:53.718280 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:53.718479 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.718483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.719576 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.719577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.720418 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.720420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.721036 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.721037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.721158 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.721689 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:53.721697 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:53.724465 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
W20260923 23:05:53.724725 14315 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:53.716466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:47947 (local address 127.5.74.62:39281)
0923 23:05:53.717212 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:53.717219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:53.717244 (+ 25us) server_negotiation.cc:407] Connection header received
0923 23:05:53.717351 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:53.717355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:53.717393 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:53.717531 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:53.718593 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.719472 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.720513 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.720926 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.721050 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.724584 (+ 3534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:47947: BlockingRecv error: recv got EOF from 127.5.74.1:47947 (error 108)
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":77,"threads_started":1}
W20260923 23:05:53.724956 14312 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:39281 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.74.62:39281: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 23:05:53.730710 14315 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:53.725232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:36687 (local address 127.5.74.62:39281)
0923 23:05:53.725270 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:53.725272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:53.725310 (+ 38us) server_negotiation.cc:407] Connection header received
0923 23:05:53.725386 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:53.725387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:53.725397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:53.725453 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:53.725710 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.726354 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.727352 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.727625 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.727700 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.730644 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:36687: BlockingRecv error: recv got EOF from 127.5.74.1:36687 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 23:05:53.730803 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.725198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:36687)
0923 23:05:53.725267 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:53.725278 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:53.725327 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:53.725455 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:53.725455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:53.725586 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:53.725629 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.725630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.726450 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.726451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.727245 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.727247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.727711 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.727712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.727793 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.728324 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:53.728330 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:53.730709 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:53.736611 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.731075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:49229)
0923 23:05:53.731123 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:53.731134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:53.731174 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:53.731344 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:53.731346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:53.731512 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:53.731556 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.731558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.732476 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.732477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.733379 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.733382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.733840 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.733841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.733911 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.734393 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:53.734398 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:53.736536 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 23:05:53.736745 14312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:05:53.736888 14315 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:53.731130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:49229 (local address 127.5.74.62:39281)
0923 23:05:53.731162 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:53.731164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:53.731172 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:53.731244 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:53.731244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:53.731254 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:53.731314 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:53.731641 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.732395 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.733470 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.733757 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.733829 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.736635 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:49229: BlockingRecv error: recv got EOF from 127.5.74.1:49229 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260923 23:05:53.813230 14316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:53.813503 14316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:53.813544 14316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:53.814741 14316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:53.814814 14316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:53.814857 14316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:05:53.816666 14316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.2.keytab
--principal=oryx/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:41671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=33127
--tserver_master_addrs=127.5.74.62:39281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:53.816960 14316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:53.817205 14316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:53.822410 14322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:53.822459 14316 server_base.cc:1034] running on GCE node
W20260923 23:05:53.822410 14321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:53.822404 14324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:53.822816 14316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:53.823074 14316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:53.824229 14316 hybrid_clock.cc:648] HybridClock initialized: now 1790204753824220 us; error 39 us; skew 500 ppm
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204753, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:53.827085 14316 init.cc:377] Logged in from keytab as oryx/127.5.74.2@KRBTEST.COM (short username oryx)
I20260923 23:05:53.827570 14316 webserver.cc:460] Webserver started at http://127.5.74.2:33127/ using document root <none> and password file <none>
I20260923 23:05:53.827761 14316 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:53.827821 14316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:53.829270 14316 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:53.829872 14331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.830049 14316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:53.830135 14316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "87b00849b73c4060b413b191b3ea0ca7"
format_stamp: "Formatted at 2026-09-23 23:05:53 on dist-test-slave-j4c5"
server_key: "0373c3cc038be42ac538ad3fb302a784"
server_key_iv: "2d0a472b51e256bb58d011e06e2def87"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.830523 14316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:53.849427 14316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:53.851910 14316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:53.852033 14316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:53.852288 14316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:53.852626 14316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:53.852663 14316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.852686 14316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:53.852699 14316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.859028 14316 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:41671
I20260923 23:05:53.859148 14444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:41671 every 8 connection(s)
I20260923 23:05:53.859452 14316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:53.867569 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.860236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:35995)
0923 23:05:53.860403 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:53.860441 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:53.860948 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:53.861200 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:53.861208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:53.861789 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:53.862009 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.862013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.862851 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.862852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.863818 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.863820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.864282 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.864283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.864426 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.865037 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:53.865047 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:53.867409 (+ 2362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":118,"threads_started":1}
W20260923 23:05:53.867568 14315 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:53.860236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:35995 (local address 127.5.74.62:39281)
0923 23:05:53.860347 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:53.860349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:53.860522 (+ 173us) server_negotiation.cc:407] Connection header received
0923 23:05:53.861049 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:53.861049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:53.861065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:53.861130 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:53.862115 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.862703 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.863924 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.864149 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.864271 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.867519 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:35995: BlockingRecv error: recv got EOF from 127.5.74.2:35995 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260923 23:05:53.867836 14445 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:39281 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.74.62:39281: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
I20260923 23:05:53.869495 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 14316
I20260923 23:05:53.869822 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:36551
--local_ip_for_outbound_sockets=127.5.74.3
--tserver_master_addrs=127.5.74.62:39281
--webserver_port=44075
--webserver_interface=127.5.74.3
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.74.3
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:53.872841 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.868098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:44619)
0923 23:05:53.868219 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:53.868228 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:53.868272 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:53.868381 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:53.868382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:53.868496 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:53.868544 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.868546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.869131 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.869132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.869708 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.869709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.870155 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.870155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.870205 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.870661 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:53.870666 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:53.872752 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260923 23:05:53.872896 14315 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:53.868177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:44619 (local address 127.5.74.62:39281)
0923 23:05:53.868226 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:53.868227 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:53.868260 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:05:53.868317 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:53.868318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:53.868326 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:53.868376 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:53.868598 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.869070 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.869779 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.870080 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.870159 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.872831 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:44619: BlockingRecv error: recv got EOF from 127.5.74.2:44619 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:53.877924 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:53.873300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:42301)
0923 23:05:53.873353 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:53.873363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:53.873433 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:53.873588 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:53.873588 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:53.873741 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:53.873781 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.873783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.874409 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.874410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.874907 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:53.874909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.875255 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.875256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:53.875307 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.875747 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:53.875752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:53.877806 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 23:05:53.877935 14315 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:53.873374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:42301 (local address 127.5.74.62:39281)
0923 23:05:53.873430 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:53.873432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:53.873441 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:53.873499 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:53.873499 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:53.873508 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:53.873581 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:53.873854 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.874320 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.874991 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:53.875175 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:53.875256 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:53.877883 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:42301: BlockingRecv error: recv got EOF from 127.5.74.2:42301 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260923 23:05:53.878193 14445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:05:53.955358 14448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:05:53.955626 14448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:05:53.955655 14448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:05:53.957077 14448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:05:53.957130 14448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:05:53.957161 14448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:05:53.958807 14448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/krb5kdc/oryx_127.5.74.3.keytab
--principal=oryx/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:36551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=44075
--tserver_master_addrs=127.5.74.62:39281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:05:53.959050 14448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:05:53.959259 14448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:05:53.964989 14456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:05:53.965003 14453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:53.965060 14448 server_base.cc:1034] running on GCE node
W20260923 23:05:53.965003 14454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:05:53.965376 14448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:05:53.965680 14448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:05:53.966830 14448 hybrid_clock.cc:648] HybridClock initialized: now 1790204753966796 us; error 51 us; skew 500 ppm
Sep 23 23:05:53 dist-test-slave-j4c5 krb5kdc[13581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204753, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:05:53.969451 14448 init.cc:377] Logged in from keytab as oryx/127.5.74.3@KRBTEST.COM (short username oryx)
I20260923 23:05:53.970041 14448 webserver.cc:460] Webserver started at http://127.5.74.3:44075/ using document root <none> and password file <none>
I20260923 23:05:53.970343 14448 fs_manager.cc:359] Metadata directory not provided
I20260923 23:05:53.970516 14448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:05:53.972123 14448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:05:53.972955 14463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:53.973155 14448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:05:53.973248 14448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "4ebe546a7ea2428ea111521d53f5c2f1"
format_stamp: "Formatted at 2026-09-23 23:05:53 on dist-test-slave-j4c5"
server_key: "af9a0e18f8e3e8e24aef9066d3c264d6"
server_key_iv: "2ad68568fac3ff6fca4bba08b16dca07"
server_key_version: "encryptionkey@0"
I20260923 23:05:53.973661 14448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:05:54.006714 14448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:05:54.009085 14448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:05:54.009253 14448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:05:54.009567 14448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:05:54.009963 14448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:05:54.010011 14448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:54.010036 14448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:05:54.010049 14448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:05:54.016098 14448 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:36551
I20260923 23:05:54.016139 14576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:36551 every 8 connection(s)
I20260923 23:05:54.016513 14448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 23:05:54.024123 14315 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:54.017217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:55307 (local address 127.5.74.62:39281)
0923 23:05:54.017302 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:54.017304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:54.017532 (+ 228us) server_negotiation.cc:407] Connection header received
0923 23:05:54.018014 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:54.018015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:54.018026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:54.018082 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:54.019155 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.019705 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.020761 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.020949 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.021008 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.024061 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:55307: BlockingRecv error: recv got EOF from 127.5.74.3:55307 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 23:05:54.024230 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:54.017186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:55307)
0923 23:05:54.017353 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:54.017389 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:54.017921 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:54.018137 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:54.018140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:54.018849 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:54.019048 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.019053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.019794 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.019794 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.020676 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.020679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.021023 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.021024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.021123 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.021668 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:54.021677 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:54.023993 (+ 2316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
W20260923 23:05:54.024493 14577 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:39281 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.74.62:39281: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
I20260923 23:05:54.026307 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 14448
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:54.029014 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:54.024721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:57209)
0923 23:05:54.024770 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:54.024781 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:54.024821 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:54.025078 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:54.025079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:54.025194 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:54.025238 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.025240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.025844 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.025845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.026345 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.026347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.026750 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.026751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.026823 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.027268 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:54.027273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:54.028965 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 23:05:54.029124 14315 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:54.024871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:57209 (local address 127.5.74.62:39281)
0923 23:05:54.024934 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:54.024936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:54.024944 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:05:54.024964 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:54.024965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:54.024972 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:54.025027 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:54.025305 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.025770 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.026453 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.026668 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.026750 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.029061 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:57209: BlockingRecv error: recv got EOF from 127.5.74.3:57209 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 23:05:54.033609 14315 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:54.029415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:40613 (local address 127.5.74.62:39281)
0923 23:05:54.029457 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:54.029459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:54.029468 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:05:54.029532 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:54.029533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:54.029541 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:54.029589 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:54.029819 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.030254 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.031004 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.031187 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.031262 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.033534 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:40613: BlockingRecv error: recv got EOF from 127.5.74.3:40613 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:05:54.033771 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:54.029353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:40613)
0923 23:05:54.029426 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:54.029437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:54.029477 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:54.029596 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:54.029597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:54.029719 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:54.029745 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.029746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.030342 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.030343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.030929 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.030930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.031263 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.031264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.031315 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.031656 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:54.031660 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:54.033708 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 23:05:54.033910 14577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:05:54.035125 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:54.026728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.5.74.62:39281)
0923 23:05:54.026835 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:54.026837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:54.026972 (+ 135us) server_negotiation.cc:407] Connection header received
0923 23:05:54.027046 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:54.027047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:54.027058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:54.027131 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:54.027704 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.028366 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.029667 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.029867 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.029954 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.031186 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:05:54.031192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:05:54.031192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:05:54.031205 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:05:54.033563 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:54.034005 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:54.034009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:54.034009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:54.034120 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:05:54.034326 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:05:54.034327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:05:54.034327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:05:54.034659 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:05:54.034797 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 23:05:54.034916 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 23:05:54.035031 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:54.742094 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:54.737185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:35515)
0923 23:05:54.737302 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:54.737318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:54.737366 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:54.737548 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:54.737549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:54.737716 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:54.737764 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.737765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.738472 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.738473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.739086 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.739088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.739469 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.739470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.739514 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.740038 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:54.740043 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:54.742019 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
W20260923 23:05:54.742249 14583 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:54.737238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:35515 (local address 127.5.74.62:39281)
0923 23:05:54.737420 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:54.737422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:54.737432 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:05:54.737456 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:54.737456 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:54.737467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:54.737537 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:54.737838 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.738377 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.739174 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.739395 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.739456 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.742161 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:35515: BlockingRecv error: recv got EOF from 127.5.74.1:35515 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":127,"threads_started":1}
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:54 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:54.883311 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:54.878606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:49955)
0923 23:05:54.878780 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:54.878793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:54.878868 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:54.879016 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:54.879017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:54.879245 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:54.879291 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.879293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.879966 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.879967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.880536 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:54.880538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.880921 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.880921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:54.880978 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.881439 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:54.881444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:54.883240 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
W20260923 23:05:54.883467 14583 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:54.878703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:49955 (local address 127.5.74.62:39281)
0923 23:05:54.878784 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:54.878786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:54.878848 (+ 62us) server_negotiation.cc:407] Connection header received
0923 23:05:54.878928 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:54.878929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:54.878942 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:54.879010 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:54.879371 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.879875 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.880622 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:54.880816 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:54.880920 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:54.883342 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:49955: BlockingRecv error: recv got EOF from 127.5.74.2:49955 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 23:05:55 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:55 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:55.039053 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:55.034394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:59323)
0923 23:05:55.034551 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:55.034568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:55.034631 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:55.034761 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:55.034761 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:55.034922 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:55.034969 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:55.034971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.035575 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.035577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:55.036176 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:55.036178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.036617 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.036618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:55.036682 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:55.037143 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:55.037147 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:55.038945 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
W20260923 23:05:55.039099 14583 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:55.034493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:59323 (local address 127.5.74.62:39281)
0923 23:05:55.034562 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:55.034564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:55.034615 (+ 51us) server_negotiation.cc:407] Connection header received
0923 23:05:55.034674 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:55.034675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:55.034686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:55.034732 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:55.035023 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.035495 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.036329 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.036529 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.036615 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:55.039030 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:59323: BlockingRecv error: recv got EOF from 127.5.74.3:59323 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 23 23:05:55 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:55 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:55.747642 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:55.742615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:33193)
0923 23:05:55.742771 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:55.742782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:55.742842 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:55.742987 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:55.742988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:55.743148 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:55.743195 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:55.743196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.743985 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.743985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:55.744566 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:55.744569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.744964 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.744965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:55.745008 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:55.745582 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:55.745587 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:55.747551 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
W20260923 23:05:55.747766 14587 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:55.742731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:33193 (local address 127.5.74.62:39281)
0923 23:05:55.742846 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:55.742848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:55.742859 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:05:55.742884 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:55.742884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:55.742896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:55.742959 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:55.743329 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.743846 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.744653 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.744867 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.744939 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:55.747668 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:33193: BlockingRecv error: recv got EOF from 127.5.74.1:33193 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
Sep 23 23:05:55 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:55 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:55.888634 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:55.883802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:34779)
0923 23:05:55.883924 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:55.883936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:55.884002 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:55.884107 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:55.884107 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:55.884296 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:55.884344 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:55.884345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.885010 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.885011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:55.885557 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:55.885559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.885971 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.885973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:55.886032 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:55.886509 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:55.886514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:55.888565 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
W20260923 23:05:55.888801 14587 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:55.883840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:34779 (local address 127.5.74.62:39281)
0923 23:05:55.883888 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:55.883891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:55.884000 (+ 109us) server_negotiation.cc:407] Connection header received
0923 23:05:55.884046 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:55.884047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:55.884057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:55.884103 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:55.884428 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.884933 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.885678 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:55.885878 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:55.885970 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:55.888727 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:34779: BlockingRecv error: recv got EOF from 127.5.74.2:34779 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 23 23:05:56 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:56 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:56.044595 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:56.039595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:58653)
0923 23:05:56.039763 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:56.039774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:56.039851 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:56.040036 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:56.040037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:56.040273 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:56.040318 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:56.040320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.040989 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.040991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:56.041530 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:56.041531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.041881 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.041882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:56.041935 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:56.042452 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:56.042457 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:56.044516 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
W20260923 23:05:56.044735 14587 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:56.039694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:58653 (local address 127.5.74.62:39281)
0923 23:05:56.039758 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:56.039761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:56.039833 (+ 72us) server_negotiation.cc:407] Connection header received
0923 23:05:56.039950 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:56.039951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:56.039964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:56.040027 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:56.040395 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.040904 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.041618 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.041808 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.041881 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:56.044639 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:58653: BlockingRecv error: recv got EOF from 127.5.74.3:58653 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 23 23:05:56 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:56 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:56.753624 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:56.748312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:43351)
0923 23:05:56.748476 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:56.748488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:56.748557 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:56.748741 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:56.748742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:56.748923 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:56.748972 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:56.748973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.749677 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.749677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:56.750271 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:56.750272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.750702 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.750703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:56.750751 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:56.751325 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:56.751329 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:56.753514 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
W20260923 23:05:56.753681 14591 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:56.748343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:43351 (local address 127.5.74.62:39281)
0923 23:05:56.748477 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:56.748484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:56.748554 (+ 70us) server_negotiation.cc:407] Connection header received
0923 23:05:56.748605 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:56.748606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:56.748616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:56.748678 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:56.749066 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.749601 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.750362 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.750582 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.750641 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:56.753600 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:43351: BlockingRecv error: recv got EOF from 127.5.74.1:43351 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
Sep 23 23:05:56 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:56 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:56.894167 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:56.889140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:34271)
0923 23:05:56.889289 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:56.889299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:56.889344 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:56.889461 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:56.889461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:56.889620 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:56.889667 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:56.889669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.890324 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.890326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:56.891061 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:56.891064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.891435 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.891436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:56.891498 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:56.892078 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:56.892082 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:56.894087 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
W20260923 23:05:56.894304 14591 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:56.889239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:34271 (local address 127.5.74.62:39281)
0923 23:05:56.889322 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:56.889325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:56.889364 (+ 39us) server_negotiation.cc:407] Connection header received
0923 23:05:56.889391 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:56.889391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:56.889402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:56.889448 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:56.889731 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.890225 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.891160 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:56.891348 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:56.891403 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:56.894225 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:34271: BlockingRecv error: recv got EOF from 127.5.74.2:34271 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 23:05:57 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:57 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:57.050222 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:57.045121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:44047)
0923 23:05:57.045264 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:57.045278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:57.045402 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:57.045552 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:57.045553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:57.045741 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:57.045787 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:57.045789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.046458 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.046460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:57.047019 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:57.047021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.047371 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.047372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:57.047431 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:57.047959 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:57.047964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:57.050108 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260923 23:05:57.050278 14591 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:57.045234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:44047 (local address 127.5.74.62:39281)
0923 23:05:57.045300 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:57.045303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:57.045397 (+ 94us) server_negotiation.cc:407] Connection header received
0923 23:05:57.045462 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:57.045463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:57.045478 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:57.045546 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:57.045872 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.046373 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.047102 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.047288 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.047370 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:57.050206 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:44047: BlockingRecv error: recv got EOF from 127.5.74.3:44047 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 23 23:05:57 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:57 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:57.759934 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:57.754333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:38915)
0923 23:05:57.754523 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:57.754542 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:57.754636 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:57.754790 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:57.754791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:57.755135 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:57.755194 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:57.755197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.756003 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.756003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:57.756733 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:57.756736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.757127 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.757128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:57.757171 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:57.757648 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:57.757653 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:57.759831 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
W20260923 23:05:57.759995 14595 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:57.754378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:38915 (local address 127.5.74.62:39281)
0923 23:05:57.754514 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:57.754517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:57.754636 (+ 119us) server_negotiation.cc:407] Connection header received
0923 23:05:57.754685 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:57.754686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:57.754700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:57.754787 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:57.755302 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.755884 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.756829 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.757046 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.757112 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:57.759922 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:38915: BlockingRecv error: recv got EOF from 127.5.74.1:38915 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":52,"threads_started":1}
Sep 23 23:05:57 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:57 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:57.901392 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:57.894749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:38739)
0923 23:05:57.894894 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:57.894910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:57.894967 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:57.895216 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:57.895218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:57.895448 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:57.895500 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:57.895502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.896721 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.896722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:57.897266 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:57.897268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.897736 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.897737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:57.897792 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:57.898328 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:57.898375 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:57.901249 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
W20260923 23:05:57.901427 14595 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:57.894861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:38739 (local address 127.5.74.62:39281)
0923 23:05:57.894927 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:57.894929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:57.894962 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:05:57.895074 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:57.895075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:57.895088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:57.895158 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:57.895811 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.896629 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.897411 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:57.897622 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:57.897688 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:57.901360 (+ 3672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:38739: BlockingRecv error: recv got EOF from 127.5.74.2:38739 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 23 23:05:58 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:58 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:58.056025 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:58.051020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:35159)
0923 23:05:58.051153 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:58.051172 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:58.051251 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:58.051545 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:58.051546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:58.051712 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:58.051759 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:58.051760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.052480 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.052481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:58.053038 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:58.053041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.053415 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.053416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:58.053462 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:58.053922 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:58.053927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:58.055953 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
W20260923 23:05:58.056281 14595 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:58.051127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:35159 (local address 127.5.74.62:39281)
0923 23:05:58.051183 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:58.051185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:58.051229 (+ 44us) server_negotiation.cc:407] Connection header received
0923 23:05:58.051351 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:58.051351 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:58.051364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:58.051522 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:58.051844 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.052381 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.053127 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.053310 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.053380 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:58.056201 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:35159: BlockingRecv error: recv got EOF from 127.5.74.3:35159 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 23 23:05:58 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:58 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:58.772452 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:58.760486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:35653)
0923 23:05:58.760614 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:58.760626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:58.760691 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:58.760861 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:58.760861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:58.761019 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:58.761065 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:58.761067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.762168 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.762169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:58.762732 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:58.762734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.763309 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.763310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:58.763360 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:58.769283 (+ 5923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:58.769290 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:58.772367 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
W20260923 23:05:58.772650 14599 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:58.760541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:35653 (local address 127.5.74.62:39281)
0923 23:05:58.760658 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:58.760660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:58.760674 (+ 14us) server_negotiation.cc:407] Connection header received
0923 23:05:58.760741 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:58.760742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:58.760756 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:58.760836 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:58.761137 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.762050 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.762822 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.763202 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.763278 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:58.772538 (+ 9260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:35653: BlockingRecv error: recv got EOF from 127.5.74.1:35653 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
Sep 23 23:05:58 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:58 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:58.907164 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:58.901998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:39993)
0923 23:05:58.902113 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:58.902125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:58.902209 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:58.902374 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:58.902375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:58.902547 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:58.902596 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:58.902597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.903276 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.903277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:58.903817 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:58.903819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.904281 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.904282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:58.904330 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:58.904877 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:58.904883 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:58.907100 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
W20260923 23:05:58.907362 14599 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:58.902075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:39993 (local address 127.5.74.62:39281)
0923 23:05:58.902130 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:58.902132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:58.902197 (+ 65us) server_negotiation.cc:407] Connection header received
0923 23:05:58.902274 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:58.902275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:58.902289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:58.902345 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:58.902676 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.903197 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.903918 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:58.904120 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:58.904209 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:58.907288 (+ 3079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:39993: BlockingRecv error: recv got EOF from 127.5.74.2:39993 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 23 23:05:59 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:59 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:59.062381 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:59.056726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:34541)
0923 23:05:59.056866 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:59.056878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:59.056944 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:59.057159 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:59.057160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:59.057454 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:59.057523 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:59.057525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.058384 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.058386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:59.059177 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:59.059179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.059634 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.059637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:59.059695 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:59.060249 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:59.060255 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:59.062303 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
W20260923 23:05:59.062529 14599 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:59.056835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:34541 (local address 127.5.74.62:39281)
0923 23:05:59.056919 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:59.056921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:59.056942 (+ 21us) server_negotiation.cc:407] Connection header received
0923 23:05:59.056997 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:59.056997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:59.057012 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:59.057089 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:59.057642 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.058252 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.059316 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.059519 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.059588 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:59.062438 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:34541: BlockingRecv error: recv got EOF from 127.5.74.3:34541 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 23:05:59 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:59 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:59.778513 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:59.773066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:55025)
0923 23:05:59.773246 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:59.773261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:59.773335 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:59.773576 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:59.773577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:59.773912 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:59.773987 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:59.773989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.774734 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.774735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:59.775290 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:59.775292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.775726 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.775727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:59.775772 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:59.776654 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:59.776660 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:59.778445 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
W20260923 23:05:59.778720 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:59.773190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:55025 (local address 127.5.74.62:39281)
0923 23:05:59.773347 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:59.773349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:59.773374 (+ 25us) server_negotiation.cc:407] Connection header received
0923 23:05:59.773432 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:59.773433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:59.773443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:59.773498 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:59.774082 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.774643 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.775441 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.775652 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.775711 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:59.778622 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:55025: BlockingRecv error: recv got EOF from 127.5.74.1:55025 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
Sep 23 23:05:59 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:05:59 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:05:59.912520 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:05:59.907646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:39435)
0923 23:05:59.907785 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 23:05:59.907796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:05:59.907844 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:05:59.908036 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:05:59.908037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:05:59.908283 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:05:59.908334 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:59.908336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.908996 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.908997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:59.909547 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:05:59.909549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.909924 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.909925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:05:59.909986 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:59.910457 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:05:59.910462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:05:59.912457 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
W20260923 23:05:59.912653 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:05:59.907848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:39435 (local address 127.5.74.62:39281)
0923 23:05:59.907916 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:05:59.907918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:05:59.907930 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:05:59.907951 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:05:59.907951 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:05:59.907961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:05:59.908029 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:05:59.908414 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.908915 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.909631 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:05:59.909841 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:05:59.909923 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:05:59.912551 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:39435: BlockingRecv error: recv got EOF from 127.5.74.2:39435 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 23 23:06:00 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:00 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:00.068063 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:00.063005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:38501)
0923 23:06:00.063164 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:00.063175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:00.063250 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:00.063397 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:00.063398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:00.063584 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:00.063631 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:00.063633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.064339 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.064341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:00.064963 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:00.064966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.065338 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.065339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:00.065396 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:00.065876 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:00.065880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:00.067950 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
W20260923 23:06:00.068115 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:00.063055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:38501 (local address 127.5.74.62:39281)
0923 23:06:00.063124 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:00.063126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:00.063236 (+ 110us) server_negotiation.cc:407] Connection header received
0923 23:06:00.063313 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:00.063314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:00.063332 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:00.063390 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:00.063710 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.064262 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.065053 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.065240 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.065338 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:00.068048 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:38501: BlockingRecv error: recv got EOF from 127.5.74.3:38501 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 23 23:06:00 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:00 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:00.783757 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:00.779023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:52441)
0923 23:06:00.779194 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:00.779207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:00.779284 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:00.779474 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:00.779475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:00.779638 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:00.779683 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:00.779684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.780411 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.780411 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:00.780972 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:00.780973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.781353 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.781354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:00.781403 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:00.781869 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:00.781874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:00.783659 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
W20260923 23:06:00.783824 14607 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:00.779088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:52441 (local address 127.5.74.62:39281)
0923 23:06:00.779240 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:00.779242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:00.779265 (+ 23us) server_negotiation.cc:407] Connection header received
0923 23:06:00.779383 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:00.779384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:00.779394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:00.779458 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:00.779742 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.780316 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.781050 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.781245 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.781309 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:00.783742 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:52441: BlockingRecv error: recv got EOF from 127.5.74.1:52441 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
Sep 23 23:06:00 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:00 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:00.918154 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:00.913127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:46879)
0923 23:06:00.913247 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:00.913260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:00.913325 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:00.913539 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:00.913540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:00.913717 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:00.913763 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:00.913765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.914466 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.914467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:00.915010 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:00.915012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.915382 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.915383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:00.915426 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:00.915960 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:00.915972 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:00.918042 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
W20260923 23:06:00.918298 14607 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:00.913200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:46879 (local address 127.5.74.62:39281)
0923 23:06:00.913388 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:00.913390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:00.913399 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:06:00.913424 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:00.913425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:00.913436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:00.913500 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:00.913871 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.914386 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.915095 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:00.915302 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:00.915363 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:00.918211 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:46879: BlockingRecv error: recv got EOF from 127.5.74.2:46879 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 23 23:06:01 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:01 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:01.073567 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:01.068669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:38169)
0923 23:06:01.068820 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:01.068830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:01.068889 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:01.069058 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:01.069059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:01.069222 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:01.069271 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:01.069272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.069881 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.069882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:01.070400 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:01.070401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.070735 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.070736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:01.070774 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:01.071310 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:01.071314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:01.073497 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
W20260923 23:06:01.073663 14607 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:01.068752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:38169 (local address 127.5.74.62:39281)
0923 23:06:01.068819 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:01.068821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:01.068888 (+ 67us) server_negotiation.cc:407] Connection header received
0923 23:06:01.068966 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:01.068967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:01.068980 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:01.069042 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:01.069330 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.069813 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.070494 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.070671 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.070726 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:01.073592 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:38169: BlockingRecv error: recv got EOF from 127.5.74.3:38169 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 23 23:06:01 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:01 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:01.789489 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:01.784461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:43297)
0923 23:06:01.784602 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:01.784614 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:01.784664 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:01.784893 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:01.784894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:01.785087 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:01.785132 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:01.785134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.785967 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.785968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:01.786591 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:01.786594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.786992 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.786993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:01.787042 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:01.787562 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:01.787567 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:01.789391 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
W20260923 23:06:01.789546 14611 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:01.784562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:43297 (local address 127.5.74.62:39281)
0923 23:06:01.784701 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:01.784703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:01.784714 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:06:01.784740 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:01.784741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:01.784755 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:01.784840 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:01.785240 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.785890 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.786680 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.786868 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.786992 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:01.789475 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:43297: BlockingRecv error: recv got EOF from 127.5.74.1:43297 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
Sep 23 23:06:01 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:01 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:01.923900 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:01.918932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:59199)
0923 23:06:01.919139 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:01.919151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:01.919226 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:01.919378 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:01.919379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:01.919564 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:01.919611 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:01.919613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.920502 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.920503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:01.921027 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:01.921028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.921383 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.921383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:01.921445 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:01.921933 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:01.921938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:01.923784 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":121,"threads_started":1}
W20260923 23:06:01.923939 14611 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:01.919047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:59199 (local address 127.5.74.62:39281)
0923 23:06:01.919137 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:01.919139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:01.919207 (+ 68us) server_negotiation.cc:407] Connection header received
0923 23:06:01.919281 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:01.919282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:01.919296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:01.919365 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:01.919701 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.920422 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.921108 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:01.921299 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:01.921366 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:01.923877 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:59199: BlockingRecv error: recv got EOF from 127.5.74.2:59199 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 23:06:02 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:02 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:02.079982 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:02.074135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:40425)
0923 23:06:02.074282 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:02.074301 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:02.074397 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:02.074720 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:02.074722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:02.074909 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:02.074955 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:02.074957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.075987 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.075988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:02.076638 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:02.076640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.077158 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.077159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:02.077287 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:02.077841 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:02.077845 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:02.079903 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
W20260923 23:06:02.080116 14611 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:02.074243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:40425 (local address 127.5.74.62:39281)
0923 23:06:02.074385 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:02.074387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:02.074398 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:06:02.074461 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:02.074462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:02.074473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:02.074532 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:02.075096 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.075729 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.076866 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.077075 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.077140 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:02.080028 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:40425: BlockingRecv error: recv got EOF from 127.5.74.3:40425 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 23 23:06:02 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:02 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:02.795109 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:02.790104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:35543)
0923 23:06:02.790252 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:02.790262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:02.790352 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:02.790499 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:02.790500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:02.790670 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:02.790710 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:02.790711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.791407 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.791408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:02.791949 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:02.791951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.792325 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.792326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:02.792388 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:02.792955 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:02.792961 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:02.795050 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260923 23:06:02.795346 14615 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:02.790170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:35543 (local address 127.5.74.62:39281)
0923 23:06:02.790365 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:02.790367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:02.790377 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:06:02.790402 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:02.790403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:02.790412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:02.790482 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:02.790789 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.791305 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.792028 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.792247 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.792300 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:02.795280 (+ 2980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:35543: BlockingRecv error: recv got EOF from 127.5.74.1:35543 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
Sep 23 23:06:02 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:02 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:02.929505 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:02.924552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:47985)
0923 23:06:02.924777 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:02.924788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:02.924893 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:02.925041 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:02.925041 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:02.925225 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:02.925270 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:02.925272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.925958 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.925959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:02.926482 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:02.926483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.926866 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.926867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:02.926923 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:02.927485 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:02.927489 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:02.929405 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
W20260923 23:06:02.929556 14615 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:02.924674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:47985 (local address 127.5.74.62:39281)
0923 23:06:02.924748 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:02.924750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:02.924875 (+ 125us) server_negotiation.cc:407] Connection header received
0923 23:06:02.924963 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:02.924964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:02.924976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:02.925033 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:02.925354 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.925855 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.926572 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:02.926783 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:02.926866 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:02.929491 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:47985: BlockingRecv error: recv got EOF from 127.5.74.2:47985 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 23:06:03 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:03 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:03.086056 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:03.080656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:42185)
0923 23:06:03.080881 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:03.080891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:03.080973 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:03.081164 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:03.081165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:03.081355 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:03.081399 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:03.081401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.082085 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.082086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:03.082884 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:03.082887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.083281 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.083282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:03.083380 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:03.083905 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:03.083910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:03.085996 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
W20260923 23:06:03.086177 14615 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:03.080742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:42185 (local address 127.5.74.62:39281)
0923 23:06:03.080831 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:03.080834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:03.080954 (+ 120us) server_negotiation.cc:407] Connection header received
0923 23:06:03.081054 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:03.081055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:03.081076 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:03.081138 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:03.081481 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.081998 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.082999 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.083195 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.083271 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:03.086103 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:42185: BlockingRecv error: recv got EOF from 127.5.74.3:42185 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 23:06:03 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:03 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:03.806619 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:03.795621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:38071)
0923 23:06:03.795805 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:03.795815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:03.795868 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:03.796146 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:03.796167 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:03.796348 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:03.796397 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:03.796399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.797077 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.797079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:03.797652 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:03.797654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.798025 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.798026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:03.798071 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:03.804646 (+ 6575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:03.804652 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:03.806556 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260923 23:06:03.806789 14619 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:03.795706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:38071 (local address 127.5.74.62:39281)
0923 23:06:03.795950 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:03.795953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:03.795977 (+ 24us) server_negotiation.cc:407] Connection header received
0923 23:06:03.796001 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:03.796001 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:03.796018 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:03.796094 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:03.796478 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.796985 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.797734 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.797952 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.798008 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:03.806697 (+ 8689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:38071: BlockingRecv error: recv got EOF from 127.5.74.1:38071 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":167,"threads_started":1}
Sep 23 23:06:03 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:03 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:03.935039 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:03.930054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:35577)
0923 23:06:03.930188 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:03.930202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:03.930291 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:03.930424 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:03.930425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:03.930606 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:03.930654 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:03.930656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.931329 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.931330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:03.931848 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:03.931849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.932269 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.932270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:03.932326 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:03.932799 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:03.932805 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:03.934975 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
W20260923 23:06:03.935149 14619 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:03.930203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:35577 (local address 127.5.74.62:39281)
0923 23:06:03.930275 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:03.930277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:03.930287 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:06:03.930353 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:03.930354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:03.930365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:03.930410 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:03.930751 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.931246 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.931933 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:03.932143 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:03.932253 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:03.935078 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:35577: BlockingRecv error: recv got EOF from 127.5.74.2:35577 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 23:06:04 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:04 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:04.091568 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:04.086587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:47667)
0923 23:06:04.086732 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:04.086742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:04.086790 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:04.086962 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:04.086963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:04.087193 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:04.087240 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:04.087241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.087959 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.087960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:04.088629 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:04.088631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.089066 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.089067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:04.089113 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:04.089544 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:04.089548 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:04.091449 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
W20260923 23:06:04.091629 14619 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:04.086709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:47667 (local address 127.5.74.62:39281)
0923 23:06:04.086800 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:04.086803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:04.086813 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:06:04.086861 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:04.086862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:04.086877 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:04.086946 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:04.087329 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.087863 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.088717 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.088921 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.088999 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:04.091562 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:47667: BlockingRecv error: recv got EOF from 127.5.74.3:47667 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 23:06:04 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:04 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:04.812337 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:04.807148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:36929)
0923 23:06:04.807408 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:04.807421 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:04.807503 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:04.807684 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:04.807686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:04.807912 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:04.807972 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:04.807975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.808866 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.808867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:04.809420 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:04.809422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.809840 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.809841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:04.809890 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:04.810371 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:04.810376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:04.812269 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":133,"threads_started":1}
W20260923 23:06:04.812505 14623 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:04.807253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:36929 (local address 127.5.74.62:39281)
0923 23:06:04.807414 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:04.807416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:04.807486 (+ 70us) server_negotiation.cc:407] Connection header received
0923 23:06:04.807574 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:04.807575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:04.807587 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:04.807640 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:04.808079 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.808761 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.809520 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.809746 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.809819 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:04.812413 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:36929: BlockingRecv error: recv got EOF from 127.5.74.1:36929 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":50,"threads_started":1}
Sep 23 23:06:04 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:04 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:04.940788 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:04.935666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:54375)
0923 23:06:04.935816 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:04.935828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:04.935904 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:04.936060 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:04.936061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:04.936239 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:04.936284 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:04.936286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.937038 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.937039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:04.937658 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:04.937661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.938040 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.938041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:04.938083 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:04.938624 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:04.938630 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:04.940728 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260923 23:06:04.940955 14623 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:04.935741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:54375 (local address 127.5.74.62:39281)
0923 23:06:04.935823 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:04.935827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:04.935895 (+ 68us) server_negotiation.cc:407] Connection header received
0923 23:06:04.935959 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:04.935960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:04.935982 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:04.936041 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:04.936370 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.936938 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.937746 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:04.937965 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:04.938029 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:04.940856 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:54375: BlockingRecv error: recv got EOF from 127.5.74.2:54375 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 23:06:05 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:05 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:05.097687 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:05.092325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:36759)
0923 23:06:05.092470 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:05.092499 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:05.092574 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:05.092730 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:05.092731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:05.092924 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:05.092971 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:05.092973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.093721 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.093722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:05.094410 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:05.094412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.094783 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.094784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:05.094840 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:05.095385 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:05.095390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:05.097568 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260923 23:06:05.097730 14623 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:05.092455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:36759 (local address 127.5.74.62:39281)
0923 23:06:05.092526 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:05.092528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:05.092554 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:06:05.092651 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:05.092652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:05.092665 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:05.092742 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:05.093091 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.093630 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.094501 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.094696 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.094781 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:05.097661 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:36759: BlockingRecv error: recv got EOF from 127.5.74.3:36759 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 23:06:05 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:05 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:05.818029 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:05.812920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:39551)
0923 23:06:05.813088 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:05.813107 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:05.813172 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:05.813310 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:05.813311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:05.813482 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:05.813534 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:05.813535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.814245 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.814246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:05.814802 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:05.814804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.815202 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.815203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:05.815250 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:05.816037 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:05.816043 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:05.817966 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
W20260923 23:06:05.818204 14627 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:05.813032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:39551 (local address 127.5.74.62:39281)
0923 23:06:05.813166 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:05.813168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:05.813177 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:06:05.813222 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:05.813222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:05.813232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:05.813297 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:05.813604 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.814152 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.814906 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.815117 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.815179 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:05.818091 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:39551: BlockingRecv error: recv got EOF from 127.5.74.1:39551 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
Sep 23 23:06:05 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:05 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:05.946398 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:05.941363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:60221)
0923 23:06:05.941566 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:05.941578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:05.941674 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:05.941855 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:05.941856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:05.942043 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:05.942094 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:05.942096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.942805 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.942806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:05.943373 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:05.943375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.943855 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.943857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:05.943936 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:05.944504 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:05.944509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:05.946326 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
W20260923 23:06:05.946552 14627 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:05.941472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:60221 (local address 127.5.74.62:39281)
0923 23:06:05.941563 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:05.941566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:05.941654 (+ 88us) server_negotiation.cc:407] Connection header received
0923 23:06:05.941742 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:05.941743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:05.941754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:05.941848 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:05.942177 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.942722 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.943465 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:05.943753 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:05.943847 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:05.946456 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:60221: BlockingRecv error: recv got EOF from 127.5.74.2:60221 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 23:06:06 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:06 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:06.103621 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:06.098297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:58267)
0923 23:06:06.098458 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:06.098469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:06.098528 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:06.098675 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:06.098676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:06.098863 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:06.098911 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:06.098913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.099611 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.099612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:06.100286 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:06.100289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.100636 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.100636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:06.100683 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:06.101500 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:06.101507 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:06.103549 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
W20260923 23:06:06.103844 14627 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:06.098397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:58267 (local address 127.5.74.62:39281)
0923 23:06:06.098458 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:06.098460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:06.098535 (+ 75us) server_negotiation.cc:407] Connection header received
0923 23:06:06.098568 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:06.098569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:06.098581 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:06.098644 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:06.098989 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.099528 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.100376 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.100563 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.100628 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:06.103733 (+ 3105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:58267: BlockingRecv error: recv got EOF from 127.5.74.3:58267 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 23 23:06:06 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:06 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:06.823824 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:06.818681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:43107)
0923 23:06:06.818879 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:06.818893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:06.818957 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:06.819094 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:06.819095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:06.819266 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:06.819355 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:06.819357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.820037 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.820038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:06.820698 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:06.820700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.821078 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.821079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:06.821121 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:06.821693 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:06.821697 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:06.823752 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
W20260923 23:06:06.824016 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:06.818806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:43107 (local address 127.5.74.62:39281)
0923 23:06:06.818964 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:06.818966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:06.818976 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:06:06.819002 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:06.819003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:06.819013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:06.819072 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:06.819428 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.819937 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.820787 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.821003 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.821061 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:06.823937 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:43107: BlockingRecv error: recv got EOF from 127.5.74.1:43107 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
Sep 23 23:06:06 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:06 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:06.951838 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:06.946929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:57517)
0923 23:06:06.947087 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:06.947098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:06.947180 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:06.947396 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:06.947397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:06.947572 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:06.947618 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:06.947619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.948370 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.948372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:06.948895 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:06.948896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.949272 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.949273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:06.949347 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:06.949867 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:06.949872 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:06.951754 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
W20260923 23:06:06.952021 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:06.947088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:57517 (local address 127.5.74.62:39281)
0923 23:06:06.947152 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:06.947154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:06.947162 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:06:06.947258 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:06.947259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:06.947273 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:06.947344 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:06.947705 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.948289 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.948985 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:06.949180 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:06.949272 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:06.951941 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:57517: BlockingRecv error: recv got EOF from 127.5.74.2:57517 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 23 23:06:07 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:07 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:07.109313 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:07.104233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:35193)
0923 23:06:07.104376 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:07.104390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:07.104452 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:07.104612 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:07.104613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:07.104784 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:07.104835 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:07.104837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.105569 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.105570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:07.106115 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:07.106117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.106527 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.106528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:07.106575 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:07.107087 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:07.107092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:07.109250 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
W20260923 23:06:07.109517 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:07.104323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:35193 (local address 127.5.74.62:39281)
0923 23:06:07.104414 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:07.104416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:07.104435 (+ 19us) server_negotiation.cc:407] Connection header received
0923 23:06:07.104497 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:07.104497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:07.104509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:07.104571 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:07.104928 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.105480 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.106211 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.106444 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.106515 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:07.109439 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:35193: BlockingRecv error: recv got EOF from 127.5.74.3:35193 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 23:06:07 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:07 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:07.829679 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:07.824455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:57257)
0923 23:06:07.824797 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:07.824817 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:07.824882 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:07.825028 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:07.825028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:07.825189 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:07.825238 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:07.825240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.825962 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.825963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:07.826503 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:07.826504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.826863 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.826864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:07.826907 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:07.827520 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:07.827525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:07.829553 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":107,"threads_started":1}
W20260923 23:06:07.829707 14635 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:07.824579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:57257 (local address 127.5.74.62:39281)
0923 23:06:07.824711 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:07.824713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:07.824895 (+ 182us) server_negotiation.cc:407] Connection header received
0923 23:06:07.824935 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:07.824936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:07.824947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:07.825007 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:07.825369 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.825880 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.826594 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.826789 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.826847 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:07.829636 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:57257: BlockingRecv error: recv got EOF from 127.5.74.1:57257 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
Sep 23 23:06:07 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:07 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:07.957608 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:07.952365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:45997)
0923 23:06:07.952542 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:07.952555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:07.952632 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:07.952808 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:07.952809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:07.952972 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:07.953019 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:07.953021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.953757 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.953758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:07.954323 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:07.954325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.954737 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.954738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:07.954787 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:07.955319 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:07.955324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:07.957526 (+ 2202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":121,"threads_started":1}
W20260923 23:06:07.957937 14635 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:07.952485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:45997 (local address 127.5.74.62:39281)
0923 23:06:07.952550 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:07.952551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:07.952627 (+ 76us) server_negotiation.cc:407] Connection header received
0923 23:06:07.952696 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:07.952697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:07.952707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:07.952761 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:07.953084 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.953599 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.954442 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:07.954636 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:07.954695 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:07.957865 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:45997: BlockingRecv error: recv got EOF from 127.5.74.2:45997 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 23 23:06:08 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:08 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.3@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:08.115692 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:08.109905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.3:52901)
0923 23:06:08.110097 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:08.110109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:08.110178 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:08.110356 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:08.110357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:08.110542 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:08.110591 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:08.110592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.111418 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.111419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:08.111981 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:08.111983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.112557 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.112558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:08.112609 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:08.113268 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:08.113273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:08.115600 (+ 2327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":112,"threads_started":1}
W20260923 23:06:08.115845 14635 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:08.109996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:52901 (local address 127.5.74.62:39281)
0923 23:06:08.110081 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:08.110083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:08.110183 (+ 100us) server_negotiation.cc:407] Connection header received
0923 23:06:08.110225 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:08.110225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:08.110239 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:08.110338 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:08.110694 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.111261 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.112091 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.112446 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.112518 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:08.115767 (+ 3249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.3:52901: BlockingRecv error: recv got EOF from 127.5.74.3:52901 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 23:06:08 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:08 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.1@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:08.840951 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:08.830501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.1:48921)
0923 23:06:08.830670 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:08.830683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:08.830754 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:08.830885 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:08.830886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:08.831044 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:08.831092 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:08.831093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.832013 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.832014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:08.832838 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:08.832841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.833299 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.833300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:08.833346 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:08.838496 (+ 5150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:08.838503 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:08.840870 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
W20260923 23:06:08.841135 14639 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:08.830611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:48921 (local address 127.5.74.62:39281)
0923 23:06:08.830750 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:08.830752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:08.830762 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:06:08.830809 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:08.830810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:08.830820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:08.830879 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:08.831226 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.831865 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.832956 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.833206 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.833273 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:08.841053 (+ 7780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.1:48921: BlockingRecv error: recv got EOF from 127.5.74.1:48921 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
Sep 23 23:06:08 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:06:08 dist-test-slave-j4c5 krb5kdc[13581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.74.2@KRBTEST.COM for oryx/127.5.74.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:06:08.964320 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:08.958262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39281 (local address 127.5.74.2:36321)
0923 23:06:08.958484 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:08.958496 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:08.958566 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:08.958873 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:08.958874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:08.959092 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:08.959157 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:08.959160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.960035 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.960036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:08.961036 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:08.961040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.961445 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.961446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:08.961490 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:08.962033 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:08.962039 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:08.964205 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.74.62:39281: Server oryx/127.5.74.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":134,"threads_started":1}
W20260923 23:06:08.964541 14639 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:08.958325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:36321 (local address 127.5.74.62:39281)
0923 23:06:08.958401 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:08.958403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:08.958572 (+ 169us) server_negotiation.cc:407] Connection header received
0923 23:06:08.958755 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:08.958756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:08.958775 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:08.958856 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:08.959272 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.959925 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.961145 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:08.961350 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:08.961410 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:08.964375 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.74.2:36321: BlockingRecv error: recv got EOF from 127.5.74.2:36321 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:06:09.026587 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 14164
I20260923 23:06:09.031308 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 14316
I20260923 23:06:09.035491 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 14448
I20260923 23:06:09.039628 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 14094
2026-09-23T23:06:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17618 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T23:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:06:09Z Disabled control of system clock
I20260923 23:06:09.068099 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:46881
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:46881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 23:06:09.155535 14644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:09.155757 14644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:09.155777 14644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:09.156937 14644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:06:09.156975 14644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:09.156987 14644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:09.157001 14644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:06:09.157012 14644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:06:09.158274 14644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:46881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:46881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:09.158486 14644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:09.158751 14644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:09.163930 14649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.163946 14650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.163970 14652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:09.163964 14644 server_base.cc:1034] running on GCE node
I20260923 23:06:09.164377 14644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:09.164618 14644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:09.165755 14644 hybrid_clock.cc:648] HybridClock initialized: now 1790204769165743 us; error 33 us; skew 500 ppm
I20260923 23:06:09.167133 14644 webserver.cc:460] Webserver started at http://127.5.74.62:43611/ using document root <none> and password file <none>
I20260923 23:06:09.167395 14644 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:09.167448 14644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:09.167610 14644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:09.168798 14644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "c4d7d82048054dc6aac1ea3b9eafd708"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "b484adb57a9d8d11d7d4256fdacd955a"
server_key_iv: "e75e58c02d7e6d602ffc3dae083296f1"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.169176 14644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "c4d7d82048054dc6aac1ea3b9eafd708"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "b484adb57a9d8d11d7d4256fdacd955a"
server_key_iv: "e75e58c02d7e6d602ffc3dae083296f1"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.170497 14644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:06:09.171268 14658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.171458 14644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:06:09.171548 14644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "c4d7d82048054dc6aac1ea3b9eafd708"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "b484adb57a9d8d11d7d4256fdacd955a"
server_key_iv: "e75e58c02d7e6d602ffc3dae083296f1"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.171623 14644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:09.181986 14644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:09.182327 14644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:09.182441 14644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:09.186545 14644 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:46881
I20260923 23:06:09.186590 14710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:46881 every 8 connection(s)
I20260923 23:06:09.186969 14644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:06:09.187698 14711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:06:09.190335 14711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708: Bootstrap starting.
I20260923 23:06:09.190958 14711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708: Neither blocks nor log segments found. Creating new log.
I20260923 23:06:09.191260 14711 log.cc:824] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708: Log is configured to *not* fsync() on all Append() calls
I20260923 23:06:09.191968 14711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708: No bootstrap required, opened a new log
I20260923 23:06:09.192848 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 14644
I20260923 23:06:09.192963 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:06:09.193147 5416 external_mini_cluster.cc:1428] Setting key 9eae879f50b7a73bfdfe0f45f0e7bf70
I20260923 23:06:09.194149 14711 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7d82048054dc6aac1ea3b9eafd708" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46881 } }
I20260923 23:06:09.194309 14711 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:06:09.194371 14711 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d7d82048054dc6aac1ea3b9eafd708, State: Initialized, Role: FOLLOWER
I20260923 23:06:09.194526 14711 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7d82048054dc6aac1ea3b9eafd708" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46881 } }
I20260923 23:06:09.194630 14711 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:06:09.194676 14711 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:06:09.194725 14711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:06:09.195261 14711 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7d82048054dc6aac1ea3b9eafd708" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46881 } }
I20260923 23:06:09.195377 14711 leader_election.cc:302] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d7d82048054dc6aac1ea3b9eafd708; no voters:
I20260923 23:06:09.195575 14711 leader_election.cc:288] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:06:09.195605 14716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:06:09.195708 14716 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [term 1 LEADER]: Becoming Leader. State: Replica: c4d7d82048054dc6aac1ea3b9eafd708, State: Running, Role: LEADER
I20260923 23:06:09.195794 14716 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7d82048054dc6aac1ea3b9eafd708" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46881 } }
I20260923 23:06:09.195854 14711 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:06:09.196134 14716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d7d82048054dc6aac1ea3b9eafd708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7d82048054dc6aac1ea3b9eafd708" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46881 } } }
I20260923 23:06:09.196346 14716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [sys.catalog]: This master's current role is: LEADER
I20260923 23:06:09.196537 14716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d7d82048054dc6aac1ea3b9eafd708. Latest consensus state: current_term: 1 leader_uuid: "c4d7d82048054dc6aac1ea3b9eafd708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7d82048054dc6aac1ea3b9eafd708" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 46881 } } }
I20260923 23:06:09.196614 14716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708 [sys.catalog]: This master's current role is: LEADER
I20260923 23:06:09.196877 14724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:06:09.197434 14724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:06:09.199194 14724 catalog_manager.cc:1347] Generated new cluster ID: 1597b43b6e2541398723464d54e34db7
I20260923 23:06:09.199304 14724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:06:09.199818 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.193744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.5.74.62:46881)
0923 23:06:09.194006 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:09.194013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:09.194035 (+ 22us) server_negotiation.cc:407] Connection header received
0923 23:06:09.194127 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:09.194132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:09.194185 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:09.194324 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:09.194553 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.196070 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.197221 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.197878 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.198022 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:09.198857 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:09.198861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:09.198861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:06:09.199085 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:09.199127 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:06:09.199142 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:09.199257 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:09.199534 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 23:06:09.199610 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260923 23:06:09.208264 14724 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:06:09.208963 14724 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:06:09.212438 14724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d7d82048054dc6aac1ea3b9eafd708: Generated new TSK 0
I20260923 23:06:09.212616 14724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:06:09.217514 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:46881
--builtin_ntp_servers=127.5.74.20:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 23:06:09.306003 14735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:09.306237 14735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:09.306274 14735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:09.307456 14735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:09.307520 14735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:09.307562 14735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:06:09.308993 14735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:09.309268 14735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:09.309585 14735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:09.315678 14741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.315682 14743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.315681 14740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:09.315951 14735 server_base.cc:1034] running on GCE node
I20260923 23:06:09.316242 14735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:09.316538 14735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:09.317723 14735 hybrid_clock.cc:648] HybridClock initialized: now 1790204769317712 us; error 33 us; skew 500 ppm
I20260923 23:06:09.319244 14735 webserver.cc:460] Webserver started at http://127.5.74.1:35919/ using document root <none> and password file <none>
I20260923 23:06:09.319526 14735 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:09.319595 14735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:09.319721 14735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:09.321297 14735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "4c46c754ab464f21ad7b9ac9633125e7"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "639171ad0df7dd71c95af7ef76195600"
server_key_iv: "36ff20f2c972510cb35943395378aa00"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.321687 14735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "4c46c754ab464f21ad7b9ac9633125e7"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "639171ad0df7dd71c95af7ef76195600"
server_key_iv: "36ff20f2c972510cb35943395378aa00"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.323254 14735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 23:06:09.324133 14749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.324414 14735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:06:09.324501 14735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "4c46c754ab464f21ad7b9ac9633125e7"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "639171ad0df7dd71c95af7ef76195600"
server_key_iv: "36ff20f2c972510cb35943395378aa00"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.324572 14735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:09.340721 14735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:09.341116 14735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:09.341274 14735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:09.341511 14735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:06:09.341862 14735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:06:09.341905 14735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.341943 14735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:06:09.341960 14735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.348326 14735 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:36689
I20260923 23:06:09.348414 14862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:36689 every 8 connection(s)
I20260923 23:06:09.348757 14735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:06:09.353159 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 14735
I20260923 23:06:09.353269 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:06:09.353529 5416 external_mini_cluster.cc:1428] Setting key 49bb5b8727ddf75be370ddc55c337c2a
I20260923 23:06:09.354607 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:46881
--builtin_ntp_servers=127.5.74.20:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 23:06:09.354877 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.349551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:55193 (local address 127.5.74.62:46881)
0923 23:06:09.349641 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:09.349644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:09.349834 (+ 190us) server_negotiation.cc:407] Connection header received
0923 23:06:09.350326 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:09.350327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:09.350337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:09.350391 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:09.351044 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.351526 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.352993 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.353313 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.353398 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:09.354490 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:09.354491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:09.354491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:06:09.354496 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:09.354510 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:06:09.354514 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:09.354561 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:09.354733 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 23:06:09.354775 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 23:06:09.355085 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.349518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:46881 (local address 127.5.74.1:55193)
0923 23:06:09.349715 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:09.349748 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:09.350223 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:09.350444 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:09.350447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:09.350455 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:09.350968 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:09.350972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.351685 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.351686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:09.352751 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:09.352754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.353410 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.353411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:09.353550 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:09.354323 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:06:09.354323 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:09.354366 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:06:09.354396 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:06:09.354582 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:06:09.354588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:06:09.354591 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:06:09.354673 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 23:06:09.354802 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260923 23:06:09.356431 14863 heartbeater.cc:347] Connected to a master server at 127.5.74.62:46881
I20260923 23:06:09.356562 14863 heartbeater.cc:464] Registering TS with master...
I20260923 23:06:09.356769 14863 heartbeater.cc:511] Master 127.5.74.62:46881 requested a full tablet report, sending...
I20260923 23:06:09.357317 14675 ts_manager.cc:194] Registered new tserver with Master: 4c46c754ab464f21ad7b9ac9633125e7 (127.5.74.1:36689)
I20260923 23:06:09.357970 14675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.1:55193
W20260923 23:06:09.438337 14866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:09.438589 14866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:09.438638 14866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:09.439851 14866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:09.439909 14866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:09.439951 14866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:06:09.441337 14866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:09.441596 14866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:09.441890 14866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:09.446925 14874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.446913 14871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.447070 14872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:09.447273 14866 server_base.cc:1034] running on GCE node
I20260923 23:06:09.447471 14866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:09.447713 14866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:09.448880 14866 hybrid_clock.cc:648] HybridClock initialized: now 1790204769448859 us; error 45 us; skew 500 ppm
I20260923 23:06:09.450167 14866 webserver.cc:460] Webserver started at http://127.5.74.2:35841/ using document root <none> and password file <none>
I20260923 23:06:09.450412 14866 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:09.450490 14866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:09.450639 14866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:09.451716 14866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "3b33efb9a96d43a1ba8f06280b6db7f4"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "5644e7661ed57da7d7cd18e99cc4f8d9"
server_key_iv: "0f4cbbb1797341f14bf4c67b3dc470dc"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.452081 14866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "3b33efb9a96d43a1ba8f06280b6db7f4"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "5644e7661ed57da7d7cd18e99cc4f8d9"
server_key_iv: "0f4cbbb1797341f14bf4c67b3dc470dc"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.453433 14866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:06:09.454247 14880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.454445 14866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:06:09.454550 14866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "3b33efb9a96d43a1ba8f06280b6db7f4"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "5644e7661ed57da7d7cd18e99cc4f8d9"
server_key_iv: "0f4cbbb1797341f14bf4c67b3dc470dc"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.454627 14866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:09.468618 14866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:09.469059 14866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:09.469218 14866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:09.469542 14866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:06:09.469959 14866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:06:09.470028 14866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.470063 14866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:06:09.470081 14866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.476291 14866 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:36017
I20260923 23:06:09.476367 14993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:36017 every 8 connection(s)
I20260923 23:06:09.476679 14866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:06:09.480197 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 14866
I20260923 23:06:09.480314 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:06:09.480505 5416 external_mini_cluster.cc:1428] Setting key 7c6ecd4c34ff578dfde732c3b6eed2f3
I20260923 23:06:09.481519 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:46881
--builtin_ntp_servers=127.5.74.20:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 23:06:09.482234 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.477411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:42775 (local address 127.5.74.62:46881)
0923 23:06:09.477489 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:09.477492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:09.477742 (+ 250us) server_negotiation.cc:407] Connection header received
0923 23:06:09.478237 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:09.478238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:09.478253 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:09.478303 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:09.478964 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.479444 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.480578 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.480830 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.480883 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:09.481933 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:09.481935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:09.481935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:06:09.481940 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:09.481956 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:06:09.481962 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:09.482027 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:09.482083 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 23:06:09.482114 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 23:06:09.482378 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.477415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:46881 (local address 127.5.74.2:42775)
0923 23:06:09.477620 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:09.477656 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:09.478152 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:09.478363 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:09.478366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:09.478374 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:09.478882 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:09.478886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.479529 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.479530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:09.480489 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:09.480492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.480909 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.480910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:09.481066 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:09.481775 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:06:09.481775 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:09.481813 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:06:09.481844 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:06:09.482028 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:06:09.482032 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:06:09.482034 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:06:09.482084 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 23:06:09.482166 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260923 23:06:09.482946 14994 heartbeater.cc:347] Connected to a master server at 127.5.74.62:46881
I20260923 23:06:09.483060 14994 heartbeater.cc:464] Registering TS with master...
I20260923 23:06:09.483243 14994 heartbeater.cc:511] Master 127.5.74.62:46881 requested a full tablet report, sending...
I20260923 23:06:09.483664 14675 ts_manager.cc:194] Registered new tserver with Master: 3b33efb9a96d43a1ba8f06280b6db7f4 (127.5.74.2:36017)
I20260923 23:06:09.484059 14675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.2:42775
W20260923 23:06:09.563535 14997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:09.563772 14997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:09.563808 14997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:09.565093 14997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:09.565166 14997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:09.565209 14997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:06:09.566579 14997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:09.566852 14997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:09.567160 14997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:09.572530 15002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.572570 15003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.572551 15005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:09.572633 14997 server_base.cc:1034] running on GCE node
I20260923 23:06:09.572978 14997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:09.573226 14997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:09.574462 14997 hybrid_clock.cc:648] HybridClock initialized: now 1790204769574453 us; error 37 us; skew 500 ppm
I20260923 23:06:09.575910 14997 webserver.cc:460] Webserver started at http://127.5.74.3:35609/ using document root <none> and password file <none>
I20260923 23:06:09.576257 14997 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:09.576327 14997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:09.576463 14997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:09.577616 14997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "983eb950b1014370a57e26422891f0c6"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "3ff05ca595b7833aeca4334723732c51"
server_key_iv: "870f57132cc4dcf5d05c2e16c057f61e"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.577975 14997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "983eb950b1014370a57e26422891f0c6"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "3ff05ca595b7833aeca4334723732c51"
server_key_iv: "870f57132cc4dcf5d05c2e16c057f61e"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.579316 14997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:06:09.580108 15011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.580322 14997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:06:09.580430 14997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "983eb950b1014370a57e26422891f0c6"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "3ff05ca595b7833aeca4334723732c51"
server_key_iv: "870f57132cc4dcf5d05c2e16c057f61e"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.580519 14997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:09.603039 14997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:09.603410 14997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:09.603571 14997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:09.603813 14997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:06:09.604245 14997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:06:09.604298 14997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.604341 14997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:06:09.604367 14997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.610283 14997 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:45511
I20260923 23:06:09.610318 15124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:45511 every 8 connection(s)
I20260923 23:06:09.610770 14997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:06:09.616595 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.611567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:54437 (local address 127.5.74.62:46881)
0923 23:06:09.611617 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:09.611620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:09.611889 (+ 269us) server_negotiation.cc:407] Connection header received
0923 23:06:09.612478 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:09.612479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:09.612490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:09.612547 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:09.613480 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.613972 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.615028 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.615220 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.615280 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:09.616280 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:09.616281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:09.616281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:06:09.616294 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:09.616308 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:06:09.616312 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:09.616360 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:09.616504 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 23:06:09.616530 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:06:09.616842 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.611571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:46881 (local address 127.5.74.3:54437)
0923 23:06:09.611768 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:09.611802 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:09.612369 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:09.612724 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:09.612728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:09.612737 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:09.613382 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:09.613387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.614082 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.614083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:09.614943 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:09.614946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:09.615366 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:09.615367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:09.615513 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:09.616121 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:06:09.616121 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:09.616169 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:06:09.616192 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:06:09.616414 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:06:09.616419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:06:09.616422 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:06:09.616498 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 23:06:09.616595 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260923 23:06:09.617388 15125 heartbeater.cc:347] Connected to a master server at 127.5.74.62:46881
I20260923 23:06:09.617478 15125 heartbeater.cc:464] Registering TS with master...
I20260923 23:06:09.617614 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 14997
I20260923 23:06:09.617681 15125 heartbeater.cc:511] Master 127.5.74.62:46881 requested a full tablet report, sending...
I20260923 23:06:09.617712 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:06:09.617924 5416 external_mini_cluster.cc:1428] Setting key 15da768fbf9da910c68e196d0959067b
I20260923 23:06:09.618103 14675 ts_manager.cc:194] Registered new tserver with Master: 983eb950b1014370a57e26422891f0c6 (127.5.74.3:45511)
I20260923 23:06:09.618559 14675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.3:54437
I20260923 23:06:09.618825 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 23:06:09.620702 14715 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:09.620264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.5.74.62:46881)
0923 23:06:09.620329 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:09.620331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:09.620401 (+ 70us) server_negotiation.cc:407] Connection header received
0923 23:06:09.620461 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:09.620462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:09.620471 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:09.620523 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:09.620620 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47706: BlockingRecv error: recv got EOF from 127.0.0.1:47706 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:06:09.620805 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 14735
I20260923 23:06:09.625921 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 14866
I20260923 23:06:09.630111 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 14997
I20260923 23:06:09.634891 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 14644
2026-09-23T23:06:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (594 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T23:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:06:09Z Disabled control of system clock
I20260923 23:06:09.662076 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35331
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:46585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:35331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 23:06:09.743971 15136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:09.744211 15136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:09.744233 15136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:09.745378 15136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:06:09.745416 15136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:09.745429 15136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:09.745484 15136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:06:09.745498 15136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:06:09.746851 15136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:46585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:35331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:09.747057 15136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:09.747304 15136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:09.752368 15144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:09.752351 15136 server_base.cc:1034] running on GCE node
W20260923 23:06:09.752336 15141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.752336 15142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:09.752821 15136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:09.753132 15136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:09.754321 15136 hybrid_clock.cc:648] HybridClock initialized: now 1790204769754309 us; error 40 us; skew 500 ppm
I20260923 23:06:09.755700 15136 webserver.cc:460] Webserver started at http://127.5.74.62:40815/ using document root <none> and password file <none>
I20260923 23:06:09.755983 15136 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:09.756059 15136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:09.756237 15136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:09.757318 15136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "75df4108e30942eb8d27158ba255476e"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "ae492c447f7200aa3b37a3decb252748"
server_key_iv: "530a8b74bf3106c7ce2b69400d3db9f9"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.757695 15136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "75df4108e30942eb8d27158ba255476e"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "ae492c447f7200aa3b37a3decb252748"
server_key_iv: "530a8b74bf3106c7ce2b69400d3db9f9"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.759028 15136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:06:09.759851 15150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.760022 15136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:06:09.760115 15136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "75df4108e30942eb8d27158ba255476e"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "ae492c447f7200aa3b37a3decb252748"
server_key_iv: "530a8b74bf3106c7ce2b69400d3db9f9"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.760248 15136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:09.763597 15136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:09.763882 15136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:09.764029 15136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:09.768208 15136 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:35331
I20260923 23:06:09.768290 15202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:35331 every 8 connection(s)
I20260923 23:06:09.768608 15136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:06:09.769241 15203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:06:09.771992 15203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e: Bootstrap starting.
I20260923 23:06:09.772635 15203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e: Neither blocks nor log segments found. Creating new log.
I20260923 23:06:09.772951 15203 log.cc:824] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:06:09.773638 15203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e: No bootstrap required, opened a new log
I20260923 23:06:09.775262 15203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75df4108e30942eb8d27158ba255476e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35331 } }
I20260923 23:06:09.775403 15203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:06:09.775424 15203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75df4108e30942eb8d27158ba255476e, State: Initialized, Role: FOLLOWER
I20260923 23:06:09.775578 15203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75df4108e30942eb8d27158ba255476e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35331 } }
I20260923 23:06:09.775652 15203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:06:09.775695 15203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:06:09.775753 15203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:06:09.776388 15203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75df4108e30942eb8d27158ba255476e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35331 } }
I20260923 23:06:09.776477 15203 leader_election.cc:302] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75df4108e30942eb8d27158ba255476e; no voters:
I20260923 23:06:09.776680 15203 leader_election.cc:288] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:06:09.776708 15206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:06:09.776825 15206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [term 1 LEADER]: Becoming Leader. State: Replica: 75df4108e30942eb8d27158ba255476e, State: Running, Role: LEADER
I20260923 23:06:09.776813 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 15136
I20260923 23:06:09.776911 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:06:09.776964 15203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:06:09.776981 15206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75df4108e30942eb8d27158ba255476e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35331 } }
I20260923 23:06:09.777099 5416 external_mini_cluster.cc:1428] Setting key 8463066e55582a80111d89f4e10f0d62
I20260923 23:06:09.777335 15208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75df4108e30942eb8d27158ba255476e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75df4108e30942eb8d27158ba255476e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35331 } } }
I20260923 23:06:09.777428 15208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [sys.catalog]: This master's current role is: LEADER
I20260923 23:06:09.778074 15217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:06:09.778170 15207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75df4108e30942eb8d27158ba255476e. Latest consensus state: current_term: 1 leader_uuid: "75df4108e30942eb8d27158ba255476e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75df4108e30942eb8d27158ba255476e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35331 } } }
I20260923 23:06:09.778232 15207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e [sys.catalog]: This master's current role is: LEADER
I20260923 23:06:09.778757 15217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:06:09.780200 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.777846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.5.74.62:35331)
0923 23:06:09.778582 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:09.778590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:09.778625 (+ 35us) server_negotiation.cc:407] Connection header received
0923 23:06:09.778765 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:09.778772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:09.778819 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:09.778971 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:09.779688 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:09.779692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:09.779692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:06:09.779696 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:09.779752 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:06:09.779767 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:09.779842 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:09.779934 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 23:06:09.779989 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":81,"threads_started":1}
I20260923 23:06:09.780426 15217 catalog_manager.cc:1347] Generated new cluster ID: 3e315d72ba7f41f2a628165f74e97dd6
I20260923 23:06:09.780476 15217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:06:09.806278 15217 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:06:09.806516 15217 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:06:09.812767 15217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75df4108e30942eb8d27158ba255476e: Generated new TSK 0
I20260923 23:06:09.813009 15217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:06:09.818252 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35331
--builtin_ntp_servers=127.5.74.20:46585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 23:06:09.899895 15227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:09.900163 15227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:09.900226 15227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:09.901489 15227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:09.901532 15227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:09.901559 15227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:06:09.902978 15227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:46585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:09.903251 15227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:09.903569 15227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:09.909797 15232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.909821 15235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:09.909797 15233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:09.910007 15227 server_base.cc:1034] running on GCE node
I20260923 23:06:09.910619 15227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:09.910904 15227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:09.912092 15227 hybrid_clock.cc:648] HybridClock initialized: now 1790204769912069 us; error 39 us; skew 500 ppm
I20260923 23:06:09.913620 15227 webserver.cc:460] Webserver started at http://127.5.74.1:36075/ using document root <none> and password file <none>
I20260923 23:06:09.913893 15227 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:09.913968 15227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:09.914125 15227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:09.915237 15227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "7cb43d04d3c54fc3945c3ac343cfe45e"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "a4b73c14cf387c10b3be99628f58e1b0"
server_key_iv: "7436f875e4957b4afa684834b15456b5"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.915618 15227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "7cb43d04d3c54fc3945c3ac343cfe45e"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "a4b73c14cf387c10b3be99628f58e1b0"
server_key_iv: "7436f875e4957b4afa684834b15456b5"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.917126 15227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:06:09.917973 15241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.918190 15227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:06:09.918304 15227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "7cb43d04d3c54fc3945c3ac343cfe45e"
format_stamp: "Formatted at 2026-09-23 23:06:09 on dist-test-slave-j4c5"
server_key: "a4b73c14cf387c10b3be99628f58e1b0"
server_key_iv: "7436f875e4957b4afa684834b15456b5"
server_key_version: "encryptionkey@0"
I20260923 23:06:09.918383 15227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:09.921964 15227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:09.922250 15227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:09.922401 15227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:09.922622 15227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:06:09.922926 15227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:06:09.922982 15227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.923022 15227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:06:09.923048 15227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:09.929400 15227 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:44763
I20260923 23:06:09.929437 15354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:44763 every 8 connection(s)
I20260923 23:06:09.929775 15227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:06:09.932946 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.930566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:44573 (local address 127.5.74.62:35331)
0923 23:06:09.930654 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:09.930656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:09.930828 (+ 172us) server_negotiation.cc:407] Connection header received
0923 23:06:09.931337 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:09.931338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:09.931347 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:09.931408 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:09.932642 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:09.932644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:09.932644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:06:09.932649 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:09.932668 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:06:09.932673 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:09.932752 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:09.932846 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 23:06:09.932863 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 23:06:09.933163 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:09.930479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35331 (local address 127.5.74.1:44573)
0923 23:06:09.930711 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:09.930745 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:09.931227 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:09.931482 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:09.931485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:09.931492 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:09.932489 (+ 997us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:06:09.932489 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:09.932518 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:06:09.932547 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:06:09.932753 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:06:09.932759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:06:09.932763 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:06:09.932857 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 23:06:09.932919 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":113,"threads_started":1}
I20260923 23:06:09.933502 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 15227
I20260923 23:06:09.933642 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:06:09.933912 5416 external_mini_cluster.cc:1428] Setting key 8e9d163ee512563a9994b348a572cb9a
I20260923 23:06:09.933913 15355 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35331
I20260923 23:06:09.934003 15355 heartbeater.cc:464] Registering TS with master...
I20260923 23:06:09.934197 15355 heartbeater.cc:511] Master 127.5.74.62:35331 requested a full tablet report, sending...
I20260923 23:06:09.934744 15167 ts_manager.cc:194] Registered new tserver with Master: 7cb43d04d3c54fc3945c3ac343cfe45e (127.5.74.1:44763)
I20260923 23:06:09.934957 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35331
--builtin_ntp_servers=127.5.74.20:46585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 23:06:10.016949 15358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:10.017184 15358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:10.017210 15358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:10.018509 15358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:10.018563 15358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:10.018591 15358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:06:10.020064 15358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:46585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:10.020377 15358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:10.020716 15358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:10.026015 15363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:10.026005 15366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:10.026103 15358 server_base.cc:1034] running on GCE node
W20260923 23:06:10.025991 15364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:10.026455 15358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:10.026715 15358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:10.027884 15358 hybrid_clock.cc:648] HybridClock initialized: now 1790204770027859 us; error 42 us; skew 500 ppm
I20260923 23:06:10.029168 15358 webserver.cc:460] Webserver started at http://127.5.74.2:33897/ using document root <none> and password file <none>
I20260923 23:06:10.029413 15358 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:10.029505 15358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:10.029661 15358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:10.030742 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "360dc8b85c4644f38b5d6dfcb2f8d5d3"
format_stamp: "Formatted at 2026-09-23 23:06:10 on dist-test-slave-j4c5"
server_key: "ebaf8cb9b54e84a356e1ff74c9ca3e01"
server_key_iv: "7820bbcb1ba12cab4f08dc76efc95d44"
server_key_version: "encryptionkey@0"
I20260923 23:06:10.031109 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "360dc8b85c4644f38b5d6dfcb2f8d5d3"
format_stamp: "Formatted at 2026-09-23 23:06:10 on dist-test-slave-j4c5"
server_key: "ebaf8cb9b54e84a356e1ff74c9ca3e01"
server_key_iv: "7820bbcb1ba12cab4f08dc76efc95d44"
server_key_version: "encryptionkey@0"
I20260923 23:06:10.032470 15358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:06:10.033442 15372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:10.033700 15358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:06:10.033838 15358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "360dc8b85c4644f38b5d6dfcb2f8d5d3"
format_stamp: "Formatted at 2026-09-23 23:06:10 on dist-test-slave-j4c5"
server_key: "ebaf8cb9b54e84a356e1ff74c9ca3e01"
server_key_iv: "7820bbcb1ba12cab4f08dc76efc95d44"
server_key_version: "encryptionkey@0"
I20260923 23:06:10.033921 15358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:10.037747 15358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:10.038026 15358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:10.038169 15358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:10.038373 15358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:06:10.038689 15358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:06:10.038749 15358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:10.038789 15358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:06:10.038815 15358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:10.044539 15358 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:36537
I20260923 23:06:10.044615 15485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:36537 every 8 connection(s)
I20260923 23:06:10.044961 15358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:06:10.048058 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:10.045767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:40825 (local address 127.5.74.62:35331)
0923 23:06:10.045847 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:10.045849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:10.046067 (+ 218us) server_negotiation.cc:407] Connection header received
0923 23:06:10.046536 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:10.046537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:10.046542 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:10.046607 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:10.047836 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:10.047838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:10.047838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:06:10.047842 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:10.047858 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:06:10.047862 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:10.047941 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:10.047990 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 23:06:10.048002 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 23:06:10.048297 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:10.045746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35331 (local address 127.5.74.2:40825)
0923 23:06:10.045929 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:10.045972 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:10.046429 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:10.046668 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:10.046671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:10.046679 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:10.047674 (+ 995us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:06:10.047674 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:10.047702 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:06:10.047730 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:06:10.047923 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:06:10.047926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:06:10.047928 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:06:10.047981 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:06:10.048029 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260923 23:06:10.048732 15486 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35331
I20260923 23:06:10.048813 15486 heartbeater.cc:464] Registering TS with master...
I20260923 23:06:10.048986 15486 heartbeater.cc:511] Master 127.5.74.62:35331 requested a full tablet report, sending...
I20260923 23:06:10.049346 15167 ts_manager.cc:194] Registered new tserver with Master: 360dc8b85c4644f38b5d6dfcb2f8d5d3 (127.5.74.2:36537)
I20260923 23:06:10.050007 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 15358
I20260923 23:06:10.050107 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:06:10.050330 5416 external_mini_cluster.cc:1428] Setting key c185a6939f64ae897ccbd55ee3e0142b
I20260923 23:06:10.051211 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35331
--builtin_ntp_servers=127.5.74.20:46585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 23:06:10.137056 15489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:10.137260 15489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:10.137283 15489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:10.138542 15489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:10.138590 15489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:10.138617 15489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:06:10.139984 15489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:46585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:10.140288 15489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:10.140618 15489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:10.145656 15497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:10.145656 15494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:10.145707 15495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:10.145833 15489 server_base.cc:1034] running on GCE node
I20260923 23:06:10.146133 15489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:10.146395 15489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:10.147585 15489 hybrid_clock.cc:648] HybridClock initialized: now 1790204770147575 us; error 37 us; skew 500 ppm
I20260923 23:06:10.149307 15489 webserver.cc:460] Webserver started at http://127.5.74.3:38397/ using document root <none> and password file <none>
I20260923 23:06:10.149734 15489 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:10.149797 15489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:10.149948 15489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:10.151075 15489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "40bc2c03e44e4a43a53a810d8eab9d4b"
format_stamp: "Formatted at 2026-09-23 23:06:10 on dist-test-slave-j4c5"
server_key: "d2ffd26c364e19d51b092f2fb390eb2a"
server_key_iv: "9cfc8499910c1acfa194ba2f1440271b"
server_key_version: "encryptionkey@0"
I20260923 23:06:10.151443 15489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "40bc2c03e44e4a43a53a810d8eab9d4b"
format_stamp: "Formatted at 2026-09-23 23:06:10 on dist-test-slave-j4c5"
server_key: "d2ffd26c364e19d51b092f2fb390eb2a"
server_key_iv: "9cfc8499910c1acfa194ba2f1440271b"
server_key_version: "encryptionkey@0"
I20260923 23:06:10.152920 15489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:06:10.153872 15503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:10.154098 15489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:06:10.154225 15489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "40bc2c03e44e4a43a53a810d8eab9d4b"
format_stamp: "Formatted at 2026-09-23 23:06:10 on dist-test-slave-j4c5"
server_key: "d2ffd26c364e19d51b092f2fb390eb2a"
server_key_iv: "9cfc8499910c1acfa194ba2f1440271b"
server_key_version: "encryptionkey@0"
I20260923 23:06:10.154361 15489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:10.158339 15489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:10.158635 15489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:10.158749 15489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:10.158931 15489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:06:10.159251 15489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:06:10.159284 15489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:10.159363 15489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:06:10.159397 15489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:10.166491 15489 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:40445
I20260923 23:06:10.166531 15616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:40445 every 8 connection(s)
I20260923 23:06:10.166934 15489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:06:10.170212 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:10.167767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:58393 (local address 127.5.74.62:35331)
0923 23:06:10.167855 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:10.167858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:10.168121 (+ 263us) server_negotiation.cc:407] Connection header received
0923 23:06:10.168683 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:10.168684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:10.168690 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:10.168750 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:10.170007 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:10.170008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:10.170008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:06:10.170012 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:10.170027 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:06:10.170031 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:10.170082 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:10.170142 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 23:06:10.170155 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 23:06:10.170406 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:10.167739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35331 (local address 127.5.74.3:58393)
0923 23:06:10.167971 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:10.168015 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:10.168581 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:10.168835 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:10.168839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:10.168846 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:10.169860 (+ 1014us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:06:10.169860 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:10.169889 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:06:10.169918 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:06:10.170079 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:06:10.170082 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:06:10.170084 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:06:10.170137 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:06:10.170185 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":115,"threads_started":1}
I20260923 23:06:10.170868 15617 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35331
I20260923 23:06:10.170957 15617 heartbeater.cc:464] Registering TS with master...
I20260923 23:06:10.171109 15617 heartbeater.cc:511] Master 127.5.74.62:35331 requested a full tablet report, sending...
I20260923 23:06:10.171543 15167 ts_manager.cc:194] Registered new tserver with Master: 40bc2c03e44e4a43a53a810d8eab9d4b (127.5.74.3:40445)
I20260923 23:06:10.176551 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 15489
I20260923 23:06:10.176682 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:06:10.176853 5416 external_mini_cluster.cc:1428] Setting key f8d5f8461c6433ff3123050599bac100
I20260923 23:06:10.177909 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 23:06:10.179909 15216 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:10.179438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.5.74.62:35331)
0923 23:06:10.179501 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:10.179503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:10.179601 (+ 98us) server_negotiation.cc:407] Connection header received
0923 23:06:10.179660 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:10.179661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:10.179665 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:10.179720 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:10.179856 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34612: BlockingRecv error: recv got EOF from 127.0.0.1:34612 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 23:06:10.180043 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 15227
I20260923 23:06:10.185317 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 15358
I20260923 23:06:10.190267 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 15489
I20260923 23:06:10.194900 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 15136
2026-09-23T23:06:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (560 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:06:10 dist-test-slave-j4c5 krb5kdc[15627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:06:10 dist-test-slave-j4c5 krb5kdc[15627](info): set up 2 sockets
Sep 23 23:06:10 dist-test-slave-j4c5 krb5kdc[15627](info): commencing operation
krb5kdc: starting...
W20260923 23:06:11.253194 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:06:11 dist-test-slave-j4c5 krb5kdc[15627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:06:11.385494 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41619
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:06:11.468214 15640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:11.468482 15640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:11.468526 15640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:11.469728 15640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:06:11.469801 15640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:11.469825 15640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:11.469849 15640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:06:11.469875 15640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:06:11.471357 15640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:41619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:11.471638 15640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:11.471869 15640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:11.477212 15645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:11.477255 15646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:11.477283 15640 server_base.cc:1034] running on GCE node
W20260923 23:06:11.477221 15648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:11.477797 15640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:11.478204 15640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:11.479408 15640 hybrid_clock.cc:648] HybridClock initialized: now 1790204771479382 us; error 59 us; skew 500 ppm
Sep 23 23:06:11 dist-test-slave-j4c5 krb5kdc[15627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204771, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:06:11.482317 15640 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:06:11.482834 15640 webserver.cc:460] Webserver started at http://127.5.74.62:43731/ using document root <none> and password file <none>
I20260923 23:06:11.483083 15640 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:11.483153 15640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:11.483309 15640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:11.484524 15640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "80714049d864452ca50cbe2d3289de50"
format_stamp: "Formatted at 2026-09-23 23:06:11 on dist-test-slave-j4c5"
server_key: "f3cd3a099969055858d10533c761806d"
server_key_iv: "1d79012b15fb8e81b0f8cd2ea63cfe91"
server_key_version: "encryptionkey@0"
I20260923 23:06:11.484894 15640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "80714049d864452ca50cbe2d3289de50"
format_stamp: "Formatted at 2026-09-23 23:06:11 on dist-test-slave-j4c5"
server_key: "f3cd3a099969055858d10533c761806d"
server_key_iv: "1d79012b15fb8e81b0f8cd2ea63cfe91"
server_key_version: "encryptionkey@0"
I20260923 23:06:11.486253 15640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:06:11.487080 15655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:11.487275 15640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:06:11.487425 15640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "80714049d864452ca50cbe2d3289de50"
format_stamp: "Formatted at 2026-09-23 23:06:11 on dist-test-slave-j4c5"
server_key: "f3cd3a099969055858d10533c761806d"
server_key_iv: "1d79012b15fb8e81b0f8cd2ea63cfe91"
server_key_version: "encryptionkey@0"
I20260923 23:06:11.487522 15640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:11.519563 15640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:11.521952 15640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:11.522133 15640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:11.526820 15640 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:41619
I20260923 23:06:11.527251 15707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:41619 every 8 connection(s)
I20260923 23:06:11.527572 15640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:06:11.528338 15708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:06:11.530797 15708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50: Bootstrap starting.
I20260923 23:06:11.531337 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 15640
I20260923 23:06:11.531471 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:06:11.531713 5416 external_mini_cluster.cc:1428] Setting key d9e71023b3432f7272fb2f19ed4baa47
I20260923 23:06:11.531908 15708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50: Neither blocks nor log segments found. Creating new log.
I20260923 23:06:11.532588 15708 log.cc:824] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50: Log is configured to *not* fsync() on all Append() calls
I20260923 23:06:11.533417 15708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50: No bootstrap required, opened a new log
I20260923 23:06:11.535702 15708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80714049d864452ca50cbe2d3289de50" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41619 } }
I20260923 23:06:11.535899 15708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:06:11.535954 15708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80714049d864452ca50cbe2d3289de50, State: Initialized, Role: FOLLOWER
I20260923 23:06:11.536130 15708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80714049d864452ca50cbe2d3289de50" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41619 } }
I20260923 23:06:11.536303 15708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:06:11.536355 15708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:06:11.536409 15708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:06:11.537124 15708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80714049d864452ca50cbe2d3289de50" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41619 } }
I20260923 23:06:11.537257 15708 leader_election.cc:302] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80714049d864452ca50cbe2d3289de50; no voters:
I20260923 23:06:11.537514 15708 leader_election.cc:288] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:06:11.537539 15713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:06:11.537662 15713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [term 1 LEADER]: Becoming Leader. State: Replica: 80714049d864452ca50cbe2d3289de50, State: Running, Role: LEADER
I20260923 23:06:11.537777 15713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80714049d864452ca50cbe2d3289de50" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41619 } }
I20260923 23:06:11.537813 15708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:06:11.538264 15714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80714049d864452ca50cbe2d3289de50. Latest consensus state: current_term: 1 leader_uuid: "80714049d864452ca50cbe2d3289de50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80714049d864452ca50cbe2d3289de50" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41619 } } }
I20260923 23:06:11.538463 15714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [sys.catalog]: This master's current role is: LEADER
I20260923 23:06:11.538681 15715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80714049d864452ca50cbe2d3289de50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80714049d864452ca50cbe2d3289de50" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41619 } } }
I20260923 23:06:11.538800 15715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50 [sys.catalog]: This master's current role is: LEADER
Sep 23 23:06:11 dist-test-slave-j4c5 krb5kdc[15627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:06:11.539078 15718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:06:11.539574 15718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:06:11.541446 15718 catalog_manager.cc:1347] Generated new cluster ID: db9e7b1f262e4dc3a5054f2bbace867f
I20260923 23:06:11.541496 15718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:06:11.544139 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:11.532482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.5.74.62:41619)
0923 23:06:11.532697 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:11.532708 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:11.532852 (+ 144us) server_negotiation.cc:407] Connection header received
0923 23:06:11.533387 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:11.533394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:11.533445 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:11.533624 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:11.534248 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.535158 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.535901 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.536566 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.536694 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:11.539668 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:11.539672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:11.539673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:06:11.539686 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:11.541993 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:11.542605 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:11.542608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:11.542609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:11.542709 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:11.542882 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:11.542882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:11.542883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:11.543167 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:11.543298 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:11.543790 (+ 492us) server_negotiation.cc:299] Negotiation successful
0923 23:06:11.543931 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":119,"threads_started":1}
I20260923 23:06:11.554775 15718 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:06:11.555557 15718 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:06:11.564134 15718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50: Generated new TSK 0
I20260923 23:06:11.564507 15718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:06:11.601050 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41619
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:06:11.692386 15734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:11.692608 15734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:11.692632 15734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:11.693816 15734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:11.693862 15734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:11.693889 15734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:06:11.695354 15734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:11.695731 15734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:11.696198 15734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:11.701332 15739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:11.701385 15742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:11.701457 15740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:11.701658 15734 server_base.cc:1034] running on GCE node
I20260923 23:06:11.701830 15734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:11.702036 15734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:11.703336 15734 hybrid_clock.cc:648] HybridClock initialized: now 1790204771703323 us; error 32 us; skew 500 ppm
Sep 23 23:06:11 dist-test-slave-j4c5 krb5kdc[15627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204771, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:06:11.706146 15734 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:06:11.706629 15734 webserver.cc:460] Webserver started at http://127.5.74.1:39689/ using document root <none> and password file <none>
I20260923 23:06:11.706878 15734 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:11.706949 15734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:11.707098 15734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:11.708245 15734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "a9e8eecb5141412cb412e810a1db797a"
format_stamp: "Formatted at 2026-09-23 23:06:11 on dist-test-slave-j4c5"
server_key: "0f6dd4df535d107c5d89af0c30b2332b"
server_key_iv: "b2a3a90d9888a98890918d48f46e11f3"
server_key_version: "encryptionkey@0"
I20260923 23:06:11.708642 15734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "a9e8eecb5141412cb412e810a1db797a"
format_stamp: "Formatted at 2026-09-23 23:06:11 on dist-test-slave-j4c5"
server_key: "0f6dd4df535d107c5d89af0c30b2332b"
server_key_iv: "b2a3a90d9888a98890918d48f46e11f3"
server_key_version: "encryptionkey@0"
I20260923 23:06:11.710043 15734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:06:11.710896 15749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:11.711061 15734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:06:11.711153 15734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "a9e8eecb5141412cb412e810a1db797a"
format_stamp: "Formatted at 2026-09-23 23:06:11 on dist-test-slave-j4c5"
server_key: "0f6dd4df535d107c5d89af0c30b2332b"
server_key_iv: "b2a3a90d9888a98890918d48f46e11f3"
server_key_version: "encryptionkey@0"
I20260923 23:06:11.711234 15734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:11.726776 15734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:11.729027 15734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:11.729202 15734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:11.729476 15734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:06:11.729804 15734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:06:11.729858 15734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:11.729895 15734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:06:11.729926 15734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:11.735896 15734 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:44699
I20260923 23:06:11.735948 15862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:44699 every 8 connection(s)
I20260923 23:06:11.736356 15734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:06:11.736763 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 15734
I20260923 23:06:11.736863 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:06:11.737104 5416 external_mini_cluster.cc:1428] Setting key 2547fef579773a5677a385261a981901
Sep 23 23:06:11 dist-test-slave-j4c5 krb5kdc[15627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204771, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:06:11.748677 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:11.737149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:35517 (local address 127.5.74.62:41619)
0923 23:06:11.737205 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:11.737208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:11.737556 (+ 348us) server_negotiation.cc:407] Connection header received
0923 23:06:11.738182 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:11.738183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:11.738195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:11.738243 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:11.739432 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.740124 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.741928 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.742158 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.742240 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:11.745044 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:11.745047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:11.745047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:06:11.745056 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:11.746617 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:11.747133 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:11.747134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:11.747135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:11.747201 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:11.747494 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:11.747495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:11.747495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:11.747627 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:11.747713 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:11.748406 (+ 693us) server_negotiation.cc:299] Negotiation successful
0923 23:06:11.748509 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:06:11.748860 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:11.737154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41619 (local address 127.5.74.1:35517)
0923 23:06:11.737367 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:11.737412 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:11.738068 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:11.738350 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:11.738354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:11.739094 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:11.739322 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:11.739328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.740281 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.740282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:11.741807 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:11.741810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.742283 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.742284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:11.742447 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:11.743114 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:11.743127 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:11.744936 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:06:11.746756 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:06:11.746764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:06:11.746767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:06:11.747031 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:06:11.747287 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:06:11.747288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:06:11.747288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:06:11.747408 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:06:11.747702 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:06:11.747709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:06:11.747992 (+ 283us) client_negotiation.cc:769] Sending connection context
0923 23:06:11.748380 (+ 388us) client_negotiation.cc:240] Negotiation successful
0923 23:06:11.748575 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
I20260923 23:06:11.749454 15863 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41619
I20260923 23:06:11.749594 15863 heartbeater.cc:464] Registering TS with master...
I20260923 23:06:11.749821 15863 heartbeater.cc:511] Master 127.5.74.62:41619 requested a full tablet report, sending...
I20260923 23:06:11.750439 15672 ts_manager.cc:194] Registered new tserver with Master: a9e8eecb5141412cb412e810a1db797a (127.5.74.1:44699)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
I20260923 23:06:11.751375 15672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:35517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:06:11.768762 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41619
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:06:11.856365 15868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:11.856588 15868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:11.856614 15868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:11.857862 15868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:11.857921 15868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:11.857951 15868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:06:11.859465 15868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:11.859747 15868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:11.860003 15868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:11.865554 15874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:11.865568 15873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:11.865931 15868 server_base.cc:1034] running on GCE node
W20260923 23:06:11.865955 15876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:11.866235 15868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:11.866518 15868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:11.867697 15868 hybrid_clock.cc:648] HybridClock initialized: now 1790204771867670 us; error 71 us; skew 500 ppm
Sep 23 23:06:11 dist-test-slave-j4c5 krb5kdc[15627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204771, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:06:11.870886 15868 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:06:11.871317 15868 webserver.cc:460] Webserver started at http://127.5.74.2:41587/ using document root <none> and password file <none>
I20260923 23:06:11.871519 15868 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:11.871563 15868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:11.871654 15868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:11.872792 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e"
format_stamp: "Formatted at 2026-09-23 23:06:11 on dist-test-slave-j4c5"
server_key: "e6d55c460f2b898e58c018aa8b44e705"
server_key_iv: "151d32e1863574b8fa1bdc55c19bcce2"
server_key_version: "encryptionkey@0"
I20260923 23:06:11.873114 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e"
format_stamp: "Formatted at 2026-09-23 23:06:11 on dist-test-slave-j4c5"
server_key: "e6d55c460f2b898e58c018aa8b44e705"
server_key_iv: "151d32e1863574b8fa1bdc55c19bcce2"
server_key_version: "encryptionkey@0"
I20260923 23:06:11.874491 15868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:06:11.875247 15883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:11.875443 15868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:06:11.875519 15868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e"
format_stamp: "Formatted at 2026-09-23 23:06:11 on dist-test-slave-j4c5"
server_key: "e6d55c460f2b898e58c018aa8b44e705"
server_key_iv: "151d32e1863574b8fa1bdc55c19bcce2"
server_key_version: "encryptionkey@0"
I20260923 23:06:11.875576 15868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:11.899520 15868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:11.901901 15868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:11.902029 15868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:11.902226 15868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:06:11.902602 15868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:06:11.902645 15868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:11.902669 15868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:06:11.902681 15868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:11.908550 15868 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:45575
I20260923 23:06:11.908622 15996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:45575 every 8 connection(s)
I20260923 23:06:11.908924 15868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:06:11.914268 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 15868
I20260923 23:06:11.914404 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:06:11.914644 5416 external_mini_cluster.cc:1428] Setting key ccff766c2501a3a472ea3280a16ecd2f
Sep 23 23:06:11 dist-test-slave-j4c5 krb5kdc[15627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204771, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:06:11.918454 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:11.909646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:55509 (local address 127.5.74.62:41619)
0923 23:06:11.909729 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:11.909731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:11.909892 (+ 161us) server_negotiation.cc:407] Connection header received
0923 23:06:11.910358 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:11.910359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:11.910368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:11.910445 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:11.911445 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.911937 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.912895 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.913091 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.913146 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:11.915601 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:11.915603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:11.915603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:06:11.915611 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:11.917069 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:11.917502 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:11.917504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:11.917504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:11.917562 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:11.917811 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:11.917812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:11.917813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:11.917934 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:11.918003 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:11.918279 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 23:06:11.918361 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 23:06:11.918668 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:11.909599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41619 (local address 127.5.74.2:55509)
0923 23:06:11.909775 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:11.909811 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:11.910281 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:11.910478 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:11.910481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:11.911139 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:11.911331 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:11.911337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.912015 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.912016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:11.912821 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:11.912823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:11.913163 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:11.913164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:11.913258 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:11.913829 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:11.913838 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:11.915504 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:06:11.917169 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:06:11.917176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:06:11.917178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:06:11.917405 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:06:11.917628 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:06:11.917629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:06:11.917629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:06:11.917743 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:06:11.918007 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:06:11.918013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:06:11.918152 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 23:06:11.918261 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 23:06:11.918432 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260923 23:06:11.919278 15997 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41619
I20260923 23:06:11.919361 15997 heartbeater.cc:464] Registering TS with master...
I20260923 23:06:11.919591 15997 heartbeater.cc:511] Master 127.5.74.62:41619 requested a full tablet report, sending...
I20260923 23:06:11.920032 15672 ts_manager.cc:194] Registered new tserver with Master: 832ca8d957ff4e5b88b439e4e7f2fa2e (127.5.74.2:45575)
I20260923 23:06:11.920470 15672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:55509
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:06:11.943718 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41619
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:06:12.026456 16002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:12.026688 16002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:12.026714 16002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:12.027901 16002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:12.027953 16002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:12.027983 16002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:06:12.029508 16002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:12.029744 16002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:12.029940 16002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:12.035079 16008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:12.035068 16010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:12.035382 16002 server_base.cc:1034] running on GCE node
W20260923 23:06:12.035096 16007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:12.035738 16002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:12.036016 16002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:12.037189 16002 hybrid_clock.cc:648] HybridClock initialized: now 1790204772037185 us; error 62 us; skew 500 ppm
Sep 23 23:06:12 dist-test-slave-j4c5 krb5kdc[15627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204772, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:06:12.040011 16002 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:06:12.040584 16002 webserver.cc:460] Webserver started at http://127.5.74.3:44507/ using document root <none> and password file <none>
I20260923 23:06:12.040791 16002 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:12.040834 16002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:12.040926 16002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:06:12.041962 16002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "c5b425af5ff7432989a5ece5a2667e78"
format_stamp: "Formatted at 2026-09-23 23:06:12 on dist-test-slave-j4c5"
server_key: "4973a042d04e6dff2d811aaec7236fe2"
server_key_iv: "58ab1656a5de79e8d9eb6ab2e11e3360"
server_key_version: "encryptionkey@0"
I20260923 23:06:12.042258 16002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "c5b425af5ff7432989a5ece5a2667e78"
format_stamp: "Formatted at 2026-09-23 23:06:12 on dist-test-slave-j4c5"
server_key: "4973a042d04e6dff2d811aaec7236fe2"
server_key_iv: "58ab1656a5de79e8d9eb6ab2e11e3360"
server_key_version: "encryptionkey@0"
I20260923 23:06:12.043524 16002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:06:12.044299 16017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:12.044492 16002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:06:12.044582 16002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "c5b425af5ff7432989a5ece5a2667e78"
format_stamp: "Formatted at 2026-09-23 23:06:12 on dist-test-slave-j4c5"
server_key: "4973a042d04e6dff2d811aaec7236fe2"
server_key_iv: "58ab1656a5de79e8d9eb6ab2e11e3360"
server_key_version: "encryptionkey@0"
I20260923 23:06:12.044658 16002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:12.070571 16002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:12.072829 16002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:12.072955 16002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:12.073148 16002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:06:12.073469 16002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:06:12.073503 16002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:12.073527 16002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:06:12.073566 16002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:12.079632 16002 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:44923
I20260923 23:06:12.079725 16130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:44923 every 8 connection(s)
I20260923 23:06:12.080057 16002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:06:12 dist-test-slave-j4c5 krb5kdc[15627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204772, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:06:12.089195 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 16002
I20260923 23:06:12.089305 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:06:12.089524 5416 external_mini_cluster.cc:1428] Setting key 63598a68fa6447d507ab3084ed0945c8
I20260923 23:06:12.090960 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.080938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:48203 (local address 127.5.74.62:41619)
0923 23:06:12.080985 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:12.080987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:12.081196 (+ 209us) server_negotiation.cc:407] Connection header received
0923 23:06:12.081715 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:12.081717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:12.081727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:12.081796 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:12.082780 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.083281 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.084534 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.084742 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.084807 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.087290 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:12.087293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:12.087293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:06:12.087302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:12.088841 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:12.089598 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:12.089600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:12.089601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:12.089706 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:12.090055 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:12.090057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:12.090057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:12.090248 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:12.090325 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:12.090779 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 23:06:12.090888 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:06:12.091130 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.080872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41619 (local address 127.5.74.3:48203)
0923 23:06:12.081041 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:12.081085 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:12.081588 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:12.081872 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:12.081876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:12.082489 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:06:12.082686 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.082690 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.083420 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.083421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.084454 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.084456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.084833 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.084834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.084958 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.085452 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:06:12.085463 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:06:12.086960 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:06:12.088948 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:06:12.088995 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:06:12.088998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:06:12.089439 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:06:12.089791 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:06:12.089807 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:06:12.089807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:06:12.089969 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:06:12.090328 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:06:12.090376 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:06:12.090602 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 23:06:12.090718 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 23:06:12.090921 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260923 23:06:12.091639 16131 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41619
I20260923 23:06:12.091809 16131 heartbeater.cc:464] Registering TS with master...
I20260923 23:06:12.092036 16131 heartbeater.cc:511] Master 127.5.74.62:41619 requested a full tablet report, sending...
I20260923 23:06:12.092577 15672 ts_manager.cc:194] Registered new tserver with Master: c5b425af5ff7432989a5ece5a2667e78 (127.5.74.3:44923)
I20260923 23:06:12.092990 15672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:48203
I20260923 23:06:12.101824 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:06:12.110363 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.103252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.5.74.62:41619)
0923 23:06:12.103422 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:12.103424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:12.103509 (+ 85us) server_negotiation.cc:407] Connection header received
0923 23:06:12.103613 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:12.103614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:12.103631 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:12.103709 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:12.104406 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.104904 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.105655 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.105856 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.105961 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.107390 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:12.107392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:12.107392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:06:12.107400 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:12.108784 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:12.109319 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:12.109320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:12.109321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:12.109389 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:12.109718 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:12.109720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:12.109720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:12.109923 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:12.110047 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:12.110158 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 23:06:12.110251 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260923 23:06:12.112470 15672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:06:12.114253 15672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:06:12.121675 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.117214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:44699 (local address 127.0.0.1:33856)
0923 23:06:12.117419 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:12.117429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:12.117475 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:12.117892 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:12.117893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:12.117894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:06:12.117986 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.117988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.119355 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.119356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.120399 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.120402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.121371 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.121373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.121439 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.121444 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:06:12.121491 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:06:12.121568 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":8,"thread_start_us":76,"threads_started":1}
I20260923 23:06:12.121766 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.117111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.5.74.3:44923)
0923 23:06:12.117610 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:12.117613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:12.117622 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:06:12.117659 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:12.117661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:12.117675 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:12.118074 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:06:12.118149 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.119224 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.120686 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.121371 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.121483 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.121538 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:12.121636 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 23:06:12.121673 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":107,"threads_started":1}
I20260923 23:06:12.122011 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.117415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.5.74.2:45575)
0923 23:06:12.117598 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:12.117601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:12.117609 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:06:12.117640 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:12.117641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:12.117689 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:12.118072 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:06:12.118155 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.119012 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.120740 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.121219 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.121327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.121384 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:12.121906 (+ 522us) server_negotiation.cc:299] Negotiation successful
0923 23:06:12.121954 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260923 23:06:12.122396 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.117402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.5.74.1:44699)
0923 23:06:12.117552 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:12.117557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:12.117568 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:06:12.117610 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:12.117613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:12.117630 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:12.117709 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:06:12.118082 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.119249 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.120488 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.121201 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.121276 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.121335 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:12.122256 (+ 921us) server_negotiation.cc:299] Negotiation successful
0923 23:06:12.122309 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260923 23:06:12.121814 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.116997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:45575 (local address 127.0.0.1:47642)
0923 23:06:12.117308 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:12.117320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:12.117376 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:12.117903 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:12.117904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:12.117907 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:06:12.117985 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.117987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.119155 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.119157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.120672 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.120674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.121331 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.121332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.121410 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.121423 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 23:06:12.121498 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 23:06:12.121574 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":61,"thread_start_us":92,"threads_started":1}
I20260923 23:06:12.121827 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.116979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:44923 (local address 127.0.0.1:45458)
0923 23:06:12.117145 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:12.117185 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:12.117252 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:12.117883 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:12.117887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:12.117890 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:06:12.117986 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.117989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.119332 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.119333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.120593 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.120596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.121503 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.121504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.121556 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.121560 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:06:12.121700 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 23:06:12.121754 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":17,"thread_start_us":85,"threads_started":1}
I20260923 23:06:12.123375 15931 tablet_service.cc:1467] Processing CreateTablet for tablet 74b233b78bb941aa99763122a717ec7c (DEFAULT_TABLE table=test-table [id=22460d131f064899943fb2250923394b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:06:12.123378 15797 tablet_service.cc:1467] Processing CreateTablet for tablet 74b233b78bb941aa99763122a717ec7c (DEFAULT_TABLE table=test-table [id=22460d131f064899943fb2250923394b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:06:12.123450 16065 tablet_service.cc:1467] Processing CreateTablet for tablet 74b233b78bb941aa99763122a717ec7c (DEFAULT_TABLE table=test-table [id=22460d131f064899943fb2250923394b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:06:12.123759 16065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b233b78bb941aa99763122a717ec7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:06:12.123759 15797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b233b78bb941aa99763122a717ec7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:06:12.123759 15931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b233b78bb941aa99763122a717ec7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:06:12.126693 16148 tablet_bootstrap.cc:492] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: Bootstrap starting.
I20260923 23:06:12.127205 16149 tablet_bootstrap.cc:492] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: Bootstrap starting.
I20260923 23:06:12.127646 16148 tablet_bootstrap.cc:654] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: Neither blocks nor log segments found. Creating new log.
I20260923 23:06:12.127830 16149 tablet_bootstrap.cc:654] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: Neither blocks nor log segments found. Creating new log.
I20260923 23:06:12.128317 16148 log.cc:824] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: Log is configured to *not* fsync() on all Append() calls
I20260923 23:06:12.128362 16149 log.cc:824] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:06:12.128734 16150 tablet_bootstrap.cc:492] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a: Bootstrap starting.
I20260923 23:06:12.129088 16148 tablet_bootstrap.cc:492] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: No bootstrap required, opened a new log
I20260923 23:06:12.129176 16148 ts_tablet_manager.cc:1397] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:06:12.129317 16150 tablet_bootstrap.cc:654] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a: Neither blocks nor log segments found. Creating new log.
I20260923 23:06:12.129762 16150 log.cc:824] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a: Log is configured to *not* fsync() on all Append() calls
I20260923 23:06:12.130302 16149 tablet_bootstrap.cc:492] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: No bootstrap required, opened a new log
I20260923 23:06:12.130385 16149 ts_tablet_manager.cc:1397] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 23:06:12.130542 16150 tablet_bootstrap.cc:492] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a: No bootstrap required, opened a new log
I20260923 23:06:12.130673 16150 ts_tablet_manager.cc:1397] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:06:12.131177 16148 raft_consensus.cc:348] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } }
I20260923 23:06:12.131400 16148 raft_consensus.cc:374] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:06:12.131475 16148 raft_consensus.cc:729] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b425af5ff7432989a5ece5a2667e78, State: Initialized, Role: FOLLOWER
I20260923 23:06:12.131650 16148 consensus_queue.cc:260] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } }
I20260923 23:06:12.132092 16148 ts_tablet_manager.cc:1428] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:06:12.132313 16131 heartbeater.cc:503] Master 127.5.74.62:41619 was elected leader, sending a full tablet report...
I20260923 23:06:12.132970 16149 raft_consensus.cc:348] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } }
I20260923 23:06:12.133179 16149 raft_consensus.cc:374] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:06:12.133241 16149 raft_consensus.cc:729] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832ca8d957ff4e5b88b439e4e7f2fa2e, State: Initialized, Role: FOLLOWER
I20260923 23:06:12.133493 16149 consensus_queue.cc:260] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } }
I20260923 23:06:12.133428 16150 raft_consensus.cc:348] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } }
I20260923 23:06:12.133615 16150 raft_consensus.cc:374] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:06:12.133685 16150 raft_consensus.cc:729] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e8eecb5141412cb412e810a1db797a, State: Initialized, Role: FOLLOWER
I20260923 23:06:12.133854 15997 heartbeater.cc:503] Master 127.5.74.62:41619 was elected leader, sending a full tablet report...
I20260923 23:06:12.133845 16150 consensus_queue.cc:260] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } }
I20260923 23:06:12.134132 16150 ts_tablet_manager.cc:1428] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:06:12.134132 15863 heartbeater.cc:503] Master 127.5.74.62:41619 was elected leader, sending a full tablet report...
I20260923 23:06:12.134569 16149 ts_tablet_manager.cc:1428] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
W20260923 23:06:12.159561 15998 tablet.cc:2369] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:06:12.237095 15864 tablet.cc:2369] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:06:12.330761 16132 tablet.cc:2369] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:06:12.391072 16154 raft_consensus.cc:482] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:06:12.391224 16154 raft_consensus.cc:504] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } }
I20260923 23:06:12.391685 16154 leader_election.cc:288] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 832ca8d957ff4e5b88b439e4e7f2fa2e (127.5.74.2:45575), a9e8eecb5141412cb412e810a1db797a (127.5.74.1:44699)
I20260923 23:06:12.395284 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.392045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:45993 (local address 127.5.74.2:45575)
0923 23:06:12.392107 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:12.392110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:12.392120 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:06:12.392240 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:12.392242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:12.392256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:12.392384 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:06:12.392504 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.393072 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.394139 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.394866 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.394939 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.394948 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:12.395197 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 23:06:12.395233 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:06:12.395279 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.391954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:45575 (local address 127.5.74.3:45993)
0923 23:06:12.392035 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:12.392052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:12.392143 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:12.392337 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:12.392338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:12.392340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:06:12.392420 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.392422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.393166 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.393167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.394027 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.394029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.394981 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.394982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.395077 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.395081 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:06:12.395152 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 23:06:12.395197 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 23:06:12.395820 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.392428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:58581 (local address 127.5.74.1:44699)
0923 23:06:12.392483 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:12.392485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:12.392493 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:06:12.392522 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:12.392523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:12.392533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:12.392576 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:06:12.392764 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.393312 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.394762 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.395292 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.395360 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.395369 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:12.395735 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 23:06:12.395770 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 23:06:12.395840 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.392098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:44699 (local address 127.5.74.3:58581)
0923 23:06:12.392323 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:12.392334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:12.392389 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:12.392601 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:06:12.392602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:06:12.392604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:06:12.392668 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.392670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.393458 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.393459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.394529 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:06:12.394532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.395397 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.395398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:06:12.395491 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.395496 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:06:12.395707 (+ 211us) client_negotiation.cc:240] Negotiation successful
0923 23:06:12.395754 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":10,"thread_start_us":172,"threads_started":1}
I20260923 23:06:12.395773 15951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b233b78bb941aa99763122a717ec7c" candidate_uuid: "c5b425af5ff7432989a5ece5a2667e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" is_pre_election: true
I20260923 23:06:12.395985 15951 raft_consensus.cc:2454] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b425af5ff7432989a5ece5a2667e78 in term 0.
I20260923 23:06:12.396245 16019 leader_election.cc:302] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 832ca8d957ff4e5b88b439e4e7f2fa2e, c5b425af5ff7432989a5ece5a2667e78; no voters:
I20260923 23:06:12.396240 15817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b233b78bb941aa99763122a717ec7c" candidate_uuid: "c5b425af5ff7432989a5ece5a2667e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e8eecb5141412cb412e810a1db797a" is_pre_election: true
I20260923 23:06:12.396371 15817 raft_consensus.cc:2454] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b425af5ff7432989a5ece5a2667e78 in term 0.
I20260923 23:06:12.396389 16154 raft_consensus.cc:2781] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:06:12.396440 16154 raft_consensus.cc:482] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:06:12.396505 16154 raft_consensus.cc:3037] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:06:12.397254 16154 raft_consensus.cc:504] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } }
I20260923 23:06:12.397393 16154 leader_election.cc:288] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [CANDIDATE]: Term 1 election: Requested vote from peers 832ca8d957ff4e5b88b439e4e7f2fa2e (127.5.74.2:45575), a9e8eecb5141412cb412e810a1db797a (127.5.74.1:44699)
I20260923 23:06:12.397599 15951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b233b78bb941aa99763122a717ec7c" candidate_uuid: "c5b425af5ff7432989a5ece5a2667e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e"
I20260923 23:06:12.397634 15817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b233b78bb941aa99763122a717ec7c" candidate_uuid: "c5b425af5ff7432989a5ece5a2667e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e8eecb5141412cb412e810a1db797a"
I20260923 23:06:12.397697 15817 raft_consensus.cc:3037] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:06:12.397688 15951 raft_consensus.cc:3037] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:06:12.398597 15951 raft_consensus.cc:2454] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b425af5ff7432989a5ece5a2667e78 in term 1.
I20260923 23:06:12.398597 15817 raft_consensus.cc:2454] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b425af5ff7432989a5ece5a2667e78 in term 1.
I20260923 23:06:12.398772 16019 leader_election.cc:302] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e8eecb5141412cb412e810a1db797a, c5b425af5ff7432989a5ece5a2667e78; no voters:
I20260923 23:06:12.398890 16154 raft_consensus.cc:2781] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:06:12.399071 16154 raft_consensus.cc:686] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 1 LEADER]: Becoming Leader. State: Replica: c5b425af5ff7432989a5ece5a2667e78, State: Running, Role: LEADER
I20260923 23:06:12.399180 16154 consensus_queue.cc:237] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } }
I20260923 23:06:12.400130 15671 catalog_manager.cc:5495] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b425af5ff7432989a5ece5a2667e78 (127.5.74.3). New cstate: current_term: 1 leader_uuid: "c5b425af5ff7432989a5ece5a2667e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b425af5ff7432989a5ece5a2667e78" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 } health_report { overall_health: UNKNOWN } } }
I20260923 23:06:12.411116 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:12.408037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.5.74.3:44923)
0923 23:06:12.408226 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:12.408229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:12.408240 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:06:12.408293 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:12.408294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:12.408315 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:12.408389 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:06:12.408617 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.409082 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.410229 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:12.410438 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:12.410505 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:12.410679 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:06:12.410787 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:06:12.410860 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:12.410936 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 23:06:12.410956 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 23:06:12.413367 15817 raft_consensus.cc:1261] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [term 1 FOLLOWER]: Refusing update from remote peer c5b425af5ff7432989a5ece5a2667e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:06:12.413383 15951 raft_consensus.cc:1261] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [term 1 FOLLOWER]: Refusing update from remote peer c5b425af5ff7432989a5ece5a2667e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:06:12.413775 16158 consensus_queue.cc:1035] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e8eecb5141412cb412e810a1db797a" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:06:12.413951 16154 consensus_queue.cc:1035] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "832ca8d957ff4e5b88b439e4e7f2fa2e" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:06:12.417742 16162 mvcc.cc:205] Tried to move back new op lower bound from 7332678747802169344 to 7332678747747737600. Current Snapshot: MvccSnapshot[applied={T|T < 7332678747802169344}]
I20260923 23:06:12.419654 15671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:06:12.419776 15671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:06:12.420751 15671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80714049d864452ca50cbe2d3289de50: Sending DeleteTablet for 3 replicas of tablet 74b233b78bb941aa99763122a717ec7c
I20260923 23:06:12.421090 16065 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b233b78bb941aa99763122a717ec7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:06:12 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:45458
I20260923 23:06:12.421270 15797 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b233b78bb941aa99763122a717ec7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:06:12 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:33856
I20260923 23:06:12.421339 16171 tablet_replica.cc:330] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: stopping tablet replica
I20260923 23:06:12.421430 16171 raft_consensus.cc:2229] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:06:12.421521 16171 raft_consensus.cc:2258] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:06:12.421633 15931 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b233b78bb941aa99763122a717ec7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:06:12 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:47642
I20260923 23:06:12.421665 16172 tablet_replica.cc:330] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a: stopping tablet replica
I20260923 23:06:12.421976 16172 raft_consensus.cc:2229] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:06:12.422120 16171 ts_tablet_manager.cc:1905] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:06:12.422178 16172 raft_consensus.cc:2258] T 74b233b78bb941aa99763122a717ec7c P a9e8eecb5141412cb412e810a1db797a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:06:12.422346 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 15734
I20260923 23:06:12.425508 16171 ts_tablet_manager.cc:1918] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:06:12.425601 16171 log.cc:1196] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/74b233b78bb941aa99763122a717ec7c
I20260923 23:06:12.426265 16171 ts_tablet_manager.cc:1939] T 74b233b78bb941aa99763122a717ec7c P c5b425af5ff7432989a5ece5a2667e78: Deleting consensus metadata
I20260923 23:06:12.426994 16173 tablet_replica.cc:330] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: stopping tablet replica
I20260923 23:06:12.427409 15659 catalog_manager.cc:4841] TS c5b425af5ff7432989a5ece5a2667e78 (127.5.74.3:44923): tablet 74b233b78bb941aa99763122a717ec7c (table test-table [id=22460d131f064899943fb2250923394b]) successfully deleted
I20260923 23:06:12.427515 16173 raft_consensus.cc:2229] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:06:12.427874 16173 raft_consensus.cc:2258] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:06:12.428603 16173 ts_tablet_manager.cc:1905] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:06:12.431087 16173 ts_tablet_manager.cc:1918] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:06:12.431180 16173 log.cc:1196] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/74b233b78bb941aa99763122a717ec7c
I20260923 23:06:12.431500 16173 ts_tablet_manager.cc:1939] T 74b233b78bb941aa99763122a717ec7c P 832ca8d957ff4e5b88b439e4e7f2fa2e: Deleting consensus metadata
W20260923 23:06:12.431516 15657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:44699 (error 108)
I20260923 23:06:12.432098 15657 catalog_manager.cc:4841] TS 832ca8d957ff4e5b88b439e4e7f2fa2e (127.5.74.2:45575): tablet 74b233b78bb941aa99763122a717ec7c (table test-table [id=22460d131f064899943fb2250923394b]) successfully deleted
W20260923 23:06:12.432343 16142 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:06:12.432026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:44699 (local address 127.0.0.1:33872)
0923 23:06:12.432090 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 23:06:12.432102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:06:12.432176 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:06:12.432289 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:44699: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 23:06:12.432463 15657 catalog_manager.cc:4584] TS a9e8eecb5141412cb412e810a1db797a (127.5.74.1:44699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74b233b78bb941aa99763122a717ec7c: Network error: Client connection negotiation failed: client connection to 127.5.74.1:44699: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 23:06:12.432633 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 15868
I20260923 23:06:12.439379 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 16002
I20260923 23:06:12.444337 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 15640
2026-09-23T23:06:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2250 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:06:12 dist-test-slave-j4c5 krb5kdc[16177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:06:12 dist-test-slave-j4c5 krb5kdc[16177](info): set up 2 sockets
Sep 23 23:06:12 dist-test-slave-j4c5 krb5kdc[16177](info): commencing operation
krb5kdc: starting...
W20260923 23:06:13.541605 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:06:13 dist-test-slave-j4c5 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:06:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.74.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.74.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/rangeradmin_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/rangeradmin_127.5.74.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.74.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.74.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/rangerlookup_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/rangerlookup_127.5.74.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.74.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.74.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/HTTP_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/HTTP_127.5.74.20@KRBTEST.COM.keytab.
I20260923 23:06:13.728763 5416 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcA05g1/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/postgres -l logfile start
2026-09-23 23:06:16.677 UTC [16207] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 23:06:16.678 UTC [16207] LOG: listening on IPv4 address "127.5.74.20", port 50193
2026-09-23 23:06:16.682 UTC [16207] LOG: listening on Unix socket "/tmp/.s.PGSQL.50193"
2026-09-23 23:06:16.717 UTC [16210] LOG: database system was shut down at 2026-09-23 23:06:14 UTC
2026-09-23 23:06:16.722 UTC [16207] LOG: database system is ready to accept connections
I20260923 23:06:17.709220 5416 mini_postgres.cc:96] Postgres bound to 50193
2026-09-23 23:06:17.715 UTC [16218] FATAL: database "slave" does not exist
127.5.74.20:50193 - accepting connections
I20260923 23:06:17.715996 5416 mini_ranger.cc:161] Starting Ranger...
I20260923 23:06:17.732138 5416 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 23:06:18.359408 5416 mini_ranger.cc:87] Created ranger Postgres database
I20260923 23:06:18.359539 5416 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-admin
2026-09-23 23:06:18,674 [I] DB FLAVOR :POSTGRES
2026-09-23 23:06:18,675 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 23:06:18,675 [I] Checking connection..
2026-09-23 23:06:18,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 23:06:18,937 [I] Checking connection passed.
2026-09-23 23:06:18,937 [I] --------- Verifying version history table ---------
2026-09-23 23:06:18,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:06:19,187 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 23:06:19,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:06:19,437 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 23:06:19,437 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 23:06:19,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 23:06:19.704 UTC [16282] WARNING: there is no transaction in progress
2026-09-23 23:06:19,711 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 23:06:19,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:06:19,959 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 23:06:19,960 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 23:06:19,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 23:06:20,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 23:06:20,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j4c5.c.gcp-upstream.internal','N') ;"
2026-09-23 23:06:20,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 23:06:20,993 [I] Table x_portal_user does not exist in database ranger
2026-09-23 23:06:20,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 23:06:21,253 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 23:06:21,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 23:06:21,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 23:06:21,766 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 23:06:21,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 23:06:22.966 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:22.987 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.008 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.026 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.045 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.069 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.143 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.152 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.169 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.182 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.194 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.213 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.225 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.233 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.241 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:23.250 UTC [16390] WARNING: there is no transaction in progress
2026-09-23 23:06:24,154 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 23:06:24,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 23:06:24,439 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 23:06:24,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 23:06:24,738 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 23:06:24,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 23:06:25,004 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 23:06:25,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 23:06:25,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 23:06:25,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j4c5.c.gcp-upstream.internal';"
2026-09-23 23:06:25,752 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 23:06:25,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 23:06:26,004 [I] DB_PATCHES have already been applied
I20260923 23:06:26.009994 5416 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcA05g1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 23:06:26.010068 5416 mini_ranger.cc:193] Using host: 127.5.74.20
I20260923 23:06:26.013326 5416 mini_ranger.cc:227] Ranger admin URL: http://127.5.74.20:39627
Sep 23, 2026 11:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 11:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 11:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 11:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 11:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 11:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 11:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 23:06:27.157842 5416 mini_ranger.cc:160] Time spent starting Ranger: real 9.442s user 0.003s sys 0.007s
Sep 23, 2026 11:06:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 11:06:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 11:06:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 11:06:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 11:06:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 23:06:50.064762 5416 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.74.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.74.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/rangerkms_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/rangerkms_127.5.74.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.74.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.74.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.74.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/HTTP_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.74.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/HTTP_127.5.74.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 23:06:50 dist-test-slave-j4c5 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204810, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 23:06:50.159541 5416 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 23:06:50.169772 5416 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 23:06:51.244788 5416 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 23:06:51.244867 5416 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-kms
I20260923 23:06:51.244880 5416 mini_ranger_kms.cc:227] Using postgres at 127.5.74.20:50193
2026-09-23 23:06:51,451 [I] DB FLAVOR :POSTGRES
2026-09-23 23:06:51,452 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 23:06:51,452 [I] Checking connection
2026-09-23 23:06:51,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 23:06:51,706 [I] connection success
2026-09-23 23:06:51,706 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 23:06:51,706 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 23:06:51,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 23:06:51,966 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 23:06:51,967 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 23:06:51,967 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 23:06:51,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:50193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 23:06:52,239 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 23:06:52.785465 5416 mini_ranger_kms.cc:310] Created kms service
I20260923 23:06:52.989409 5416 mini_ranger_kms.cc:326] Created kudu user
I20260923 23:06:53.036310 5416 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 23:06:53 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204810, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:06:53.576916 5416 mini_ranger_kms.cc:384] Added ranger policy
I20260923 23:06:53.577044 5416 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 23:06:53.577067 5416 mini_ranger_kms.cc:240] Using host: 127.5.74.20
I20260923 23:06:53.580641 5416 mini_ranger_kms.cc:276] Ranger KMS PID: 16571
I20260923 23:06:53.580785 5416 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.74.20:52875
Sep 23, 2026 11:06:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 11:06:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 11:06:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 11:06:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 11:06:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 11:06:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 23:06:55.914889 5416 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.755s user 0.005s sys 0.184s
I20260923 23:06:55.915004 5416 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 23:06:55.915093 5416 mini_ranger_kms.cc:401] 127.5.74.20:52875/kms/v1/keys
Sep 23 23:06:55 dist-test-slave-j4c5 krb5kdc[16177](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790204815, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.74.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 23:06:56 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204815, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.74.20@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
Sep 23 23:06:56 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204815, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.74.20@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
Sep 23 23:06:59 dist-test-slave-j4c5 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:06:59.221334 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:36635
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:36635
--ranger_config_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:52875/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:06:59.314375 16621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:06:59.314615 16621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:06:59.314638 16621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:06:59.315881 16621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:06:59.315943 16621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:06:59.315972 16621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:06:59.315986 16621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:06:59.315999 16621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:06:59.317987 16621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.74.20:52875/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:36635
--ranger_config_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:06:59.318266 16621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:06:59.318535 16621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:06:59.326192 16626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:59.326274 16627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:06:59.326192 16629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:06:59.326192 16621 server_base.cc:1034] running on GCE node
I20260923 23:06:59.326802 16621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:06:59.327205 16621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:06:59.328433 16621 hybrid_clock.cc:648] HybridClock initialized: now 1790204819328414 us; error 28 us; skew 500 ppm
Sep 23 23:06:59 dist-test-slave-j4c5 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204819, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:06:59.335939 16621 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:06:59.336577 16621 webserver.cc:460] Webserver started at http://127.5.74.62:35111/ using document root <none> and password file <none>
I20260923 23:06:59.336894 16621 fs_manager.cc:359] Metadata directory not provided
I20260923 23:06:59.336956 16621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:06:59.337100 16621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:06:59 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204819, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:06:59.399339 16621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "46e1d1095e8e4103a4fe53ab7b6375ed"
format_stamp: "Formatted at 2026-09-23 23:06:59 on dist-test-slave-j4c5"
server_key: "750e1b2cbe7ed2c1a89bb4c6d5f4f255"
server_key_iv: "ee391710445e0de3ad4314af45354fc5"
server_key_version: "kuduclusterkey@0"
I20260923 23:06:59.399773 16621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "46e1d1095e8e4103a4fe53ab7b6375ed"
format_stamp: "Formatted at 2026-09-23 23:06:59 on dist-test-slave-j4c5"
server_key: "750e1b2cbe7ed2c1a89bb4c6d5f4f255"
server_key_iv: "ee391710445e0de3ad4314af45354fc5"
server_key_version: "kuduclusterkey@0"
I20260923 23:06:59.401319 16621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:06:59.413450 16636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:06:59.413635 16621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:06:59.413723 16621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "46e1d1095e8e4103a4fe53ab7b6375ed"
format_stamp: "Formatted at 2026-09-23 23:06:59 on dist-test-slave-j4c5"
server_key: "750e1b2cbe7ed2c1a89bb4c6d5f4f255"
server_key_iv: "ee391710445e0de3ad4314af45354fc5"
server_key_version: "kuduclusterkey@0"
I20260923 23:06:59.413848 16621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:06:59.436803 16621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:06:59.438958 16621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:06:59.439144 16621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:06:59.444849 16621 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:36635
I20260923 23:06:59.444880 16688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:36635 every 8 connection(s)
I20260923 23:06:59.445333 16621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:06:59.446044 16689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:06:59.448457 16689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed: Bootstrap starting.
I20260923 23:06:59.448798 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 16621
I20260923 23:06:59.448954 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:06:59.449105 16689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed: Neither blocks nor log segments found. Creating new log.
I20260923 23:06:59.449142 5416 external_mini_cluster.cc:1428] Setting key 5f2431069454f8eb82b19eecffded87f
I20260923 23:06:59.449522 16689 log.cc:824] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed: Log is configured to *not* fsync() on all Append() calls
I20260923 23:06:59.450356 16689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed: No bootstrap required, opened a new log
I20260923 23:06:59.452700 16689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e1d1095e8e4103a4fe53ab7b6375ed" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36635 } }
I20260923 23:06:59.452996 16689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:06:59.453053 16689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e1d1095e8e4103a4fe53ab7b6375ed, State: Initialized, Role: FOLLOWER
I20260923 23:06:59.453212 16689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e1d1095e8e4103a4fe53ab7b6375ed" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36635 } }
I20260923 23:06:59.453294 16689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:06:59.453329 16689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:06:59.453375 16689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:06:59.454346 16689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e1d1095e8e4103a4fe53ab7b6375ed" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36635 } }
I20260923 23:06:59.454536 16689 leader_election.cc:302] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46e1d1095e8e4103a4fe53ab7b6375ed; no voters:
I20260923 23:06:59.454751 16689 leader_election.cc:288] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:06:59.456046 16694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:06:59.456260 16694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [term 1 LEADER]: Becoming Leader. State: Replica: 46e1d1095e8e4103a4fe53ab7b6375ed, State: Running, Role: LEADER
I20260923 23:06:59.456386 16689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:06:59.456357 16694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e1d1095e8e4103a4fe53ab7b6375ed" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36635 } }
Sep 23 23:06:59 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:06:59.457029 16696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46e1d1095e8e4103a4fe53ab7b6375ed. Latest consensus state: current_term: 1 leader_uuid: "46e1d1095e8e4103a4fe53ab7b6375ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e1d1095e8e4103a4fe53ab7b6375ed" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36635 } } }
I20260923 23:06:59.457283 16694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46e1d1095e8e4103a4fe53ab7b6375ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e1d1095e8e4103a4fe53ab7b6375ed" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 36635 } } }
I20260923 23:06:59.457440 16696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [sys.catalog]: This master's current role is: LEADER
I20260923 23:06:59.457455 16694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed [sys.catalog]: This master's current role is: LEADER
I20260923 23:06:59.458389 16704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:06:59.458909 16704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:06:59.459244 16689 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 23:06:59.461694 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:06:59.449730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.5.74.62:36635)
0923 23:06:59.450035 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0923 23:06:59.450042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:06:59.450066 (+ 24us) server_negotiation.cc:407] Connection header received
0923 23:06:59.450535 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:06:59.450542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:06:59.450589 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:06:59.450736 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:06:59.451368 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:59.452664 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:59.453685 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:06:59.454121 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:06:59.454248 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:06:59.457294 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:06:59.457300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:06:59.457301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:06:59.457321 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:06:59.459991 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:59.460438 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:59.460441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:59.460442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:59.460537 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:06:59.460843 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:06:59.460844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:06:59.460844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:06:59.461093 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:06:59.461268 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 23:06:59.461342 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 23:06:59.461462 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":83,"threads_started":1}
I20260923 23:06:59.462530 16704 catalog_manager.cc:1347] Generated new cluster ID: aae6867aa17f486fa933ba90562c5252
I20260923 23:06:59.462582 16704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:06:59.478897 16704 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:06:59.480008 16704 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:06:59.490306 16704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed: Generated new TSK 0
I20260923 23:06:59.490620 16704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:07:00.316677 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:52875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:36635
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:00.455379 16734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:00.455677 16734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:00.455717 16734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:00.457970 16734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:00.458079 16734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:00.458151 16734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:07:00.460538 16734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.74.20:52875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:36635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:00.460963 16734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:00.461265 16734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:00.467954 16742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:00.467970 16740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:00.467979 16739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:00.468613 16734 server_base.cc:1034] running on GCE node
I20260923 23:07:00.468837 16734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:00.469139 16734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:00.470299 16734 hybrid_clock.cc:648] HybridClock initialized: now 1790204820470280 us; error 33 us; skew 500 ppm
Sep 23 23:07:00 dist-test-slave-j4c5 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:00.473634 16734 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:07:00.474150 16734 webserver.cc:460] Webserver started at http://127.5.74.1:40677/ using document root <none> and password file <none>
I20260923 23:07:00.474464 16734 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:00.474531 16734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:00.474678 16734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:07:00 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:00.489176 16734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "0fcef08ad72a45a7a3f1eab2dffc2450"
format_stamp: "Formatted at 2026-09-23 23:07:00 on dist-test-slave-j4c5"
server_key: "3446f4af645f2c4602d71206ca1cd669"
server_key_iv: "f579a76c31efd1fe4692e278861847be"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:00.489741 16734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "0fcef08ad72a45a7a3f1eab2dffc2450"
format_stamp: "Formatted at 2026-09-23 23:07:00 on dist-test-slave-j4c5"
server_key: "3446f4af645f2c4602d71206ca1cd669"
server_key_iv: "f579a76c31efd1fe4692e278861847be"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:00.491693 16734 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 23:07:00.508309 16751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:00.508592 16734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260923 23:07:00.508745 16734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "0fcef08ad72a45a7a3f1eab2dffc2450"
format_stamp: "Formatted at 2026-09-23 23:07:00 on dist-test-slave-j4c5"
server_key: "3446f4af645f2c4602d71206ca1cd669"
server_key_iv: "f579a76c31efd1fe4692e278861847be"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:00.508940 16734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:00.522784 16734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:00.525383 16734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:00.525631 16734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:00.526201 16734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:00.526679 16734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:00.526778 16734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:00.526909 16734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:00.526999 16734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:00.535882 16734 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:41527
I20260923 23:07:00.536281 16864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:41527 every 8 connection(s)
I20260923 23:07:00.537765 16734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:07:00.546548 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 16734
I20260923 23:07:00.546662 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:07:00.546855 5416 external_mini_cluster.cc:1428] Setting key 1e6cde854e75066c28fd382ce036fc43
Sep 23 23:07:00 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:00.561483 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:00.538616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36635 (local address 127.5.74.1:45985)
0923 23:07:00.540412 (+ 1796us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:00.540469 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:00.541227 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:00.541611 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:00.541616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:00.541873 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:00.542175 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:00.542182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:00.543230 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:00.543231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:00.544663 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:00.544666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:00.545395 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:00.545396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:00.545548 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:00.550653 (+ 5105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:00.550667 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:00.552841 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:00.559464 (+ 6623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:00.559479 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:00.559483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:00.559902 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:00.560226 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:00.560227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:00.560227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:00.560412 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:00.560729 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:00.560738 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:00.560946 (+ 208us) client_negotiation.cc:769] Sending connection context
0923 23:07:00.561048 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 23:07:00.561223 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1752,"thread_start_us":87,"threads_started":1}
I20260923 23:07:00.562120 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:00.538627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:45985 (local address 127.5.74.62:36635)
0923 23:07:00.538741 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:00.538743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:00.540594 (+ 1851us) server_negotiation.cc:407] Connection header received
0923 23:07:00.541406 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:00.541407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:00.541422 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:00.541476 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:00.542295 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:00.543114 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:00.544767 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:00.545078 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:00.545169 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:00.552969 (+ 7800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:00.552972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:00.552973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:00.552982 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:00.559308 (+ 6326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:00.560000 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:00.560002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:00.560003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:00.560074 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:00.560491 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:00.560492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:00.560493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:00.560643 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:00.560703 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:00.561919 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0923 23:07:00.562016 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260923 23:07:00.563035 16865 heartbeater.cc:347] Connected to a master server at 127.5.74.62:36635
I20260923 23:07:00.563134 16865 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:00.563340 16865 heartbeater.cc:511] Master 127.5.74.62:36635 requested a full tablet report, sending...
I20260923 23:07:00.563885 16653 ts_manager.cc:194] Registered new tserver with Master: 0fcef08ad72a45a7a3f1eab2dffc2450 (127.5.74.1:41527)
I20260923 23:07:00.564824 16653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:45985
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:07:00.608997 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:52875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:36635
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:07:00 dist-test-slave-j4c5 krb5kdc[16177](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790204820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 23:07:00.751204 16871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:00.751611 16871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:00.751693 16871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:00.753669 16871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:00.753931 16871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:00.754031 16871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:07:00.757287 16871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.74.20:52875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:36635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:00.757807 16871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:00.758138 16871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:00.765436 16879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:00.765964 16881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:00.766235 16878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:00.768657 16871 server_base.cc:1034] running on GCE node
I20260923 23:07:00.769075 16871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:00.769380 16871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:00.776023 16871 hybrid_clock.cc:648] HybridClock initialized: now 1790204820775985 us; error 50 us; skew 500 ppm
Sep 23 23:07:00 dist-test-slave-j4c5 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:00.779995 16871 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:07:00.780694 16871 webserver.cc:460] Webserver started at http://127.5.74.2:40123/ using document root <none> and password file <none>
I20260923 23:07:00.781031 16871 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:00.781255 16871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:00.781510 16871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:07:00 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:00.801767 16871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40"
format_stamp: "Formatted at 2026-09-23 23:07:00 on dist-test-slave-j4c5"
server_key: "37be5838174589e1f5b09f552bfa36a4"
server_key_iv: "d3c141cc556015e111fe686db8a0409a"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:00.802331 16871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40"
format_stamp: "Formatted at 2026-09-23 23:07:00 on dist-test-slave-j4c5"
server_key: "37be5838174589e1f5b09f552bfa36a4"
server_key_iv: "d3c141cc556015e111fe686db8a0409a"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:00.813028 16871 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260923 23:07:00.837612 16889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:00.837901 16871 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260923 23:07:00.838056 16871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40"
format_stamp: "Formatted at 2026-09-23 23:07:00 on dist-test-slave-j4c5"
server_key: "37be5838174589e1f5b09f552bfa36a4"
server_key_iv: "d3c141cc556015e111fe686db8a0409a"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:00.838234 16871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:00.849733 16871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:00.852267 16871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:00.852473 16871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:00.852737 16871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:00.853152 16871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:00.853221 16871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:00.853279 16871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:00.853322 16871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:00.863600 16871 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:44601
I20260923 23:07:00.864233 16871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:07:00.865767 17002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:44601 every 8 connection(s)
I20260923 23:07:00.871502 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 16871
I20260923 23:07:00.871646 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:07:00.871863 5416 external_mini_cluster.cc:1428] Setting key 1d9472123d6fa3cbdf9ab57f01d01c8e
Sep 23 23:07:00 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:00.882735 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:00.865216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:35881 (local address 127.5.74.62:36635)
0923 23:07:00.865281 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:00.865283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:00.865644 (+ 361us) server_negotiation.cc:407] Connection header received
0923 23:07:00.866403 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:00.866405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:00.866416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:00.866486 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:00.867339 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:00.868271 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:00.869979 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:00.870340 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:00.870387 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:00.878791 (+ 8404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:00.878794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:00.878794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:00.878804 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:00.880791 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:00.881387 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:00.881388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:00.881389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:00.881466 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:00.881799 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:00.881801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:00.881801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:00.881953 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:00.882018 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:00.882482 (+ 464us) server_negotiation.cc:299] Negotiation successful
0923 23:07:00.882590 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
I20260923 23:07:00.883625 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:00.865078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36635 (local address 127.5.74.2:35881)
0923 23:07:00.865515 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:00.865558 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:00.866283 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:00.866619 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:00.866624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:00.866907 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:00.867217 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:00.867223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:00.868396 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:00.868397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:00.869863 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:00.869866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:00.870508 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:00.870509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:00.870676 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:00.874662 (+ 3986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:00.874675 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:00.876582 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:00.880901 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:00.880914 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:00.880917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:00.881280 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:00.881551 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:00.881552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:00.881552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:00.881715 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:00.882070 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:00.882078 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:00.882291 (+ 213us) client_negotiation.cc:769] Sending connection context
0923 23:07:00.883140 (+ 849us) client_negotiation.cc:240] Negotiation successful
0923 23:07:00.883337 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":352,"threads_started":1}
I20260923 23:07:00.884105 17003 heartbeater.cc:347] Connected to a master server at 127.5.74.62:36635
I20260923 23:07:00.884265 17003 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:00.884480 17003 heartbeater.cc:511] Master 127.5.74.62:36635 requested a full tablet report, sending...
I20260923 23:07:00.884907 16653 ts_manager.cc:194] Registered new tserver with Master: 3b2a00bca5d046e3a3b2b20fc0d48e40 (127.5.74.2:44601)
I20260923 23:07:00.885402 16653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:35881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:07:00.917114 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:52875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:36635
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:01.016902 17008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:01.017320 17008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:01.017416 17008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:01.019579 17008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:01.019728 17008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:01.019825 17008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:07:01.022264 17008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.74.20:52875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:36635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:01.022814 17008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:01.023177 17008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:01.030702 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:01.031157 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:01.031837 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:01.032727 17008 server_base.cc:1034] running on GCE node
I20260923 23:07:01.032997 17008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:01.033309 17008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:01.034497 17008 hybrid_clock.cc:648] HybridClock initialized: now 1790204821034473 us; error 42 us; skew 500 ppm
Sep 23 23:07:01 dist-test-slave-j4c5 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204821, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:01.037616 17008 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:07:01.038280 17008 webserver.cc:460] Webserver started at http://127.5.74.3:34309/ using document root <none> and password file <none>
I20260923 23:07:01.038530 17008 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:01.038604 17008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:01.038746 17008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:07:01 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204821, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:01.049886 17008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "efb9a23482f94715afd82e59ee492420"
format_stamp: "Formatted at 2026-09-23 23:07:01 on dist-test-slave-j4c5"
server_key: "ad30d0ec75e76a5d951752acc90f721c"
server_key_iv: "e1584e0df468529913268137a1afdcfd"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:01.050468 17008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "efb9a23482f94715afd82e59ee492420"
format_stamp: "Formatted at 2026-09-23 23:07:01 on dist-test-slave-j4c5"
server_key: "ad30d0ec75e76a5d951752acc90f721c"
server_key_iv: "e1584e0df468529913268137a1afdcfd"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:01.052557 17008 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 23:07:01.068341 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:01.068589 17008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 23:07:01.068676 17008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "efb9a23482f94715afd82e59ee492420"
format_stamp: "Formatted at 2026-09-23 23:07:01 on dist-test-slave-j4c5"
server_key: "ad30d0ec75e76a5d951752acc90f721c"
server_key_iv: "e1584e0df468529913268137a1afdcfd"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:01.068770 17008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:01.094666 17008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:01.096912 17008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:01.097091 17008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:01.097373 17008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:01.097774 17008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:01.097827 17008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:01.097867 17008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:01.097894 17008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:01.105490 17008 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:44221
I20260923 23:07:01.105989 17008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:07:01.107167 17136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:44221 every 8 connection(s)
I20260923 23:07:01.114926 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 17008
I20260923 23:07:01.115036 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:07:01.115216 5416 external_mini_cluster.cc:1428] Setting key 871afac65fcd4077bf3d7886e3255836
Sep 23 23:07:01 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204821, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:01.126942 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.107873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:33359 (local address 127.5.74.62:36635)
0923 23:07:01.107929 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:01.107931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:01.112563 (+ 4632us) server_negotiation.cc:407] Connection header received
0923 23:07:01.113559 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:01.113561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:01.113575 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:01.113607 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:01.114642 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.115400 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.118566 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.118877 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.118929 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.122184 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:01.122186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:01.122186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:01.122195 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:01.124207 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:01.124805 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:01.124807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:01.124807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:01.124873 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:01.125179 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:01.125180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:01.125180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:01.125325 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:01.126288 (+ 963us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:01.126722 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 23:07:01.126834 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:07:01.132450 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.107880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:36635 (local address 127.5.74.3:33359)
0923 23:07:01.112354 (+ 4474us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:01.112431 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:01.113402 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:01.113916 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:01.113922 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:01.114212 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:01.114516 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.114521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.116215 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.116216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.118446 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.118451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.119029 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.119030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.119178 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.120336 (+ 1158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:01.120347 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:01.122051 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:01.124315 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:01.124325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:01.124328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:01.124688 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:01.124945 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:01.124946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:01.124946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:01.125100 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:01.126328 (+ 1228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:01.126336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:01.126550 (+ 214us) client_negotiation.cc:769] Sending connection context
0923 23:07:01.127089 (+ 539us) client_negotiation.cc:240] Negotiation successful
0923 23:07:01.132138 (+ 5049us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4404,"thread_start_us":94,"threads_started":1}
I20260923 23:07:01.132894 17137 heartbeater.cc:347] Connected to a master server at 127.5.74.62:36635
I20260923 23:07:01.133000 17137 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:01.133216 17137 heartbeater.cc:511] Master 127.5.74.62:36635 requested a full tablet report, sending...
I20260923 23:07:01.133584 16653 ts_manager.cc:194] Registered new tserver with Master: efb9a23482f94715afd82e59ee492420 (127.5.74.3:44221)
I20260923 23:07:01.134093 16653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:33359
I20260923 23:07:01.142863 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:07:01.161432 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.148612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.5.74.62:36635)
0923 23:07:01.148678 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:01.148681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:01.148908 (+ 227us) server_negotiation.cc:407] Connection header received
0923 23:07:01.149450 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:01.149452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:01.149468 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:01.149515 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:01.150425 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.151155 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.152669 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.152942 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.152988 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.155632 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:01.155634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:01.155634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:01.155644 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:01.159616 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:01.160483 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:01.160485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:01.160485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:01.160568 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:01.160786 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:01.160786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:01.160787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:01.160926 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:01.161002 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:01.161230 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 23:07:01.161315 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:07:01.165001 16653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:07:01.167532 16653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:07:01.188187 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.172647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:41527 (local address 127.0.0.1:60234)
0923 23:07:01.175792 (+ 3145us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:01.175800 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:01.175835 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:01.178185 (+ 2350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:01.178189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:01.178192 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:01.178293 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.178295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.182306 (+ 4011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.182308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.184094 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.184099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.187811 (+ 3712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.187812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.187915 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.187929 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:07:01.188002 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 23:07:01.188060 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3131,"thread_start_us":64,"threads_started":1}
I20260923 23:07:01.189177 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.172392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:44221 (local address 127.0.0.1:48596)
0923 23:07:01.175570 (+ 3178us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:01.175620 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:01.175685 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:01.178379 (+ 2694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:01.178379 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:01.178381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:01.178422 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.178423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.184487 (+ 6064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.184488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.185583 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.185587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.188938 (+ 3351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.188940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.189014 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.189018 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:07:01.189067 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:07:01.189108 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3143,"thread_start_us":78,"threads_started":1}
I20260923 23:07:01.192540 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.173090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.5.74.1:41527)
0923 23:07:01.176443 (+ 3353us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:01.176447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:01.176458 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:07:01.176491 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:01.176493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:01.176509 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:01.176586 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:01.179588 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.182187 (+ 2599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.186816 (+ 4629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.187632 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.187711 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.187775 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:01.190938 (+ 3163us) server_negotiation.cc:299] Negotiation successful
0923 23:07:01.190995 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3316,"thread_start_us":54,"threads_started":1}
I20260923 23:07:01.193451 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.172781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.5.74.3:44221)
0923 23:07:01.175957 (+ 3176us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:01.175981 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:01.175992 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:07:01.176032 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:01.176034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:01.176051 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:01.178477 (+ 2426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:01.178563 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.184391 (+ 5828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.185737 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.188754 (+ 3017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.188825 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.188901 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:01.191503 (+ 2602us) server_negotiation.cc:299] Negotiation successful
0923 23:07:01.191553 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3140,"thread_start_us":75,"threads_started":1}
I20260923 23:07:01.194639 16799 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd9d399e6e247e0b97e634cc225bb93 (DEFAULT_TABLE table=test-table [id=e5e9dcde80ee4252b47333ae95f845a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:07:01.195045 16799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd9d399e6e247e0b97e634cc225bb93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:01.202024 17071 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd9d399e6e247e0b97e634cc225bb93 (DEFAULT_TABLE table=test-table [id=e5e9dcde80ee4252b47333ae95f845a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:07:01.202455 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd9d399e6e247e0b97e634cc225bb93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:01.214841 17156 tablet_bootstrap.cc:492] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450: Bootstrap starting.
I20260923 23:07:01.216104 17156 tablet_bootstrap.cc:654] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:01.222262 17157 tablet_bootstrap.cc:492] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: Bootstrap starting.
I20260923 23:07:01.222936 17156 log.cc:824] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:01.223486 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.173292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:44601 (local address 127.0.0.1:46206)
0923 23:07:01.176662 (+ 3370us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:01.176673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:01.178096 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:01.212322 (+ 34226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:01.212323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:01.212337 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:01.212415 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.212418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.216486 (+ 4068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.216488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.217716 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.217723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.223181 (+ 5458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.223183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.223276 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.223284 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 23:07:01.223322 (+ 38us) client_negotiation.cc:240] Negotiation successful
0923 23:07:01.223397 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2934,"thread_start_us":192,"threads_started":1}
I20260923 23:07:01.224937 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.173411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.5.74.2:44601)
0923 23:07:01.211994 (+ 38583us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:01.212001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:01.212019 (+ 18us) server_negotiation.cc:407] Connection header received
0923 23:07:01.212074 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:01.212080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:01.212112 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:01.212500 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:01.212587 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.213756 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.220243 (+ 6487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.221130 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.224604 (+ 3474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.224745 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:01.224785 (+ 40us) server_negotiation.cc:299] Negotiation successful
0923 23:07:01.224842 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38531,"thread_start_us":105,"threads_started":1}
I20260923 23:07:01.226712 16937 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd9d399e6e247e0b97e634cc225bb93 (DEFAULT_TABLE table=test-table [id=e5e9dcde80ee4252b47333ae95f845a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:07:01.227134 16937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd9d399e6e247e0b97e634cc225bb93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:01.227411 17157 tablet_bootstrap.cc:654] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:01.227803 17157 log.cc:824] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:01.228809 17156 tablet_bootstrap.cc:492] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450: No bootstrap required, opened a new log
I20260923 23:07:01.228950 17156 ts_tablet_manager.cc:1397] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20260923 23:07:01.232221 17156 raft_consensus.cc:348] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } }
I20260923 23:07:01.232602 17156 raft_consensus.cc:374] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:01.232692 17156 raft_consensus.cc:729] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcef08ad72a45a7a3f1eab2dffc2450, State: Initialized, Role: FOLLOWER
I20260923 23:07:01.232904 17156 consensus_queue.cc:260] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } }
I20260923 23:07:01.233420 17156 ts_tablet_manager.cc:1428] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 23:07:01.233593 16865 heartbeater.cc:503] Master 127.5.74.62:36635 was elected leader, sending a full tablet report...
I20260923 23:07:01.237628 17157 tablet_bootstrap.cc:492] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: No bootstrap required, opened a new log
I20260923 23:07:01.237726 17157 ts_tablet_manager.cc:1397] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20260923 23:07:01.240466 17157 raft_consensus.cc:348] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } }
I20260923 23:07:01.240762 17157 raft_consensus.cc:374] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:01.240846 17157 raft_consensus.cc:729] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb9a23482f94715afd82e59ee492420, State: Initialized, Role: FOLLOWER
I20260923 23:07:01.241062 17157 consensus_queue.cc:260] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } }
I20260923 23:07:01.241441 17157 ts_tablet_manager.cc:1428] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 23:07:01.241621 17137 heartbeater.cc:503] Master 127.5.74.62:36635 was elected leader, sending a full tablet report...
I20260923 23:07:01.242072 17160 tablet_bootstrap.cc:492] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: Bootstrap starting.
I20260923 23:07:01.243379 17160 tablet_bootstrap.cc:654] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:01.243811 17160 log.cc:824] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:01.244729 17160 tablet_bootstrap.cc:492] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: No bootstrap required, opened a new log
I20260923 23:07:01.244808 17160 ts_tablet_manager.cc:1397] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:07:01.247074 17160 raft_consensus.cc:348] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } }
I20260923 23:07:01.247292 17160 raft_consensus.cc:374] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:01.247331 17160 raft_consensus.cc:729] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2a00bca5d046e3a3b2b20fc0d48e40, State: Initialized, Role: FOLLOWER
I20260923 23:07:01.247501 17160 consensus_queue.cc:260] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } }
I20260923 23:07:01.247805 17160 ts_tablet_manager.cc:1428] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:07:01.247944 17003 heartbeater.cc:503] Master 127.5.74.62:36635 was elected leader, sending a full tablet report...
Sep 23 23:07:01 dist-test-slave-j4c5 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
W20260923 23:07:01.291028 16866 tablet.cc:2369] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:07:01.309651 17161 raft_consensus.cc:482] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:07:01.309813 17161 raft_consensus.cc:504] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } }
I20260923 23:07:01.310217 17161 leader_election.cc:288] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb9a23482f94715afd82e59ee492420 (127.5.74.3:44221), 3b2a00bca5d046e3a3b2b20fc0d48e40 (127.5.74.2:44601)
I20260923 23:07:01.315419 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.310680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:38305 (local address 127.5.74.2:44601)
0923 23:07:01.310771 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:01.310774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:01.311867 (+ 1093us) server_negotiation.cc:407] Connection header received
0923 23:07:01.311983 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:01.311985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:01.311995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:01.312027 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:01.312312 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.313006 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.314458 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.315001 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.315050 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.315063 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:01.315280 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 23:07:01.315330 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 23:07:01.315711 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.310390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:44601 (local address 127.5.74.1:38305)
0923 23:07:01.311795 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:01.311810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:01.311901 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:01.312104 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:01.312106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:01.312107 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:01.312219 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.312222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.313126 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.313128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.314337 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.314341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.315127 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.315128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.315198 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.315202 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:07:01.315577 (+ 375us) client_negotiation.cc:240] Negotiation successful
0923 23:07:01.315625 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1370,"thread_start_us":132,"threads_started":1}
I20260923 23:07:01.316454 16957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9d399e6e247e0b97e634cc225bb93" candidate_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" is_pre_election: true
I20260923 23:07:01.316630 16957 raft_consensus.cc:2454] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fcef08ad72a45a7a3f1eab2dffc2450 in term 0.
I20260923 23:07:01.317243 16753 leader_election.cc:302] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcef08ad72a45a7a3f1eab2dffc2450, 3b2a00bca5d046e3a3b2b20fc0d48e40; no voters:
I20260923 23:07:01.317399 17161 raft_consensus.cc:2781] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:07:01.317467 17161 raft_consensus.cc:482] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:07:01.317499 17161 raft_consensus.cc:3037] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:01.318569 17161 raft_consensus.cc:504] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } }
I20260923 23:07:01.318727 17161 leader_election.cc:288] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [CANDIDATE]: Term 1 election: Requested vote from peers efb9a23482f94715afd82e59ee492420 (127.5.74.3:44221), 3b2a00bca5d046e3a3b2b20fc0d48e40 (127.5.74.2:44601)
I20260923 23:07:01.318961 16957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9d399e6e247e0b97e634cc225bb93" candidate_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40"
I20260923 23:07:01.319048 16957 raft_consensus.cc:3037] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:01.320084 16957 raft_consensus.cc:2454] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fcef08ad72a45a7a3f1eab2dffc2450 in term 1.
I20260923 23:07:01.320358 16753 leader_election.cc:302] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcef08ad72a45a7a3f1eab2dffc2450, 3b2a00bca5d046e3a3b2b20fc0d48e40; no voters:
I20260923 23:07:01.320466 17161 raft_consensus.cc:2781] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:07:01.320623 17161 raft_consensus.cc:686] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [term 1 LEADER]: Becoming Leader. State: Replica: 0fcef08ad72a45a7a3f1eab2dffc2450, State: Running, Role: LEADER
I20260923 23:07:01.320772 17161 consensus_queue.cc:237] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } }
I20260923 23:07:01.322089 16653 catalog_manager.cc:5495] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fcef08ad72a45a7a3f1eab2dffc2450 (127.5.74.1). New cstate: current_term: 1 leader_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 41527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 } health_report { overall_health: UNKNOWN } } }
I20260923 23:07:01.324301 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.310395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:44221 (local address 127.5.74.1:45777)
0923 23:07:01.315950 (+ 5555us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:01.315978 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:01.316063 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:01.316265 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:01.316266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:01.316267 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:01.316295 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.316296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.322020 (+ 5724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.322022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.322998 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:01.323002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.324027 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.324028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:01.324088 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.324092 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:07:01.324138 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:07:01.324207 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5540,"thread_start_us":129,"threads_started":1}
I20260923 23:07:01.324440 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.310725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:45777 (local address 127.5.74.3:44221)
0923 23:07:01.310805 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:01.310807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:01.316028 (+ 5221us) server_negotiation.cc:407] Connection header received
0923 23:07:01.316140 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:01.316141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:01.316174 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:01.316204 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:01.316359 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.317010 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.323112 (+ 6102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.323662 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.323716 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.323729 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:01.324291 (+ 562us) server_negotiation.cc:299] Negotiation successful
0923 23:07:01.324347 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 23:07:01.325577 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9d399e6e247e0b97e634cc225bb93" candidate_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb9a23482f94715afd82e59ee492420" is_pre_election: true
I20260923 23:07:01.325743 17091 raft_consensus.cc:2454] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fcef08ad72a45a7a3f1eab2dffc2450 in term 0.
I20260923 23:07:01.325990 17090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9d399e6e247e0b97e634cc225bb93" candidate_uuid: "0fcef08ad72a45a7a3f1eab2dffc2450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb9a23482f94715afd82e59ee492420"
I20260923 23:07:01.326078 17090 raft_consensus.cc:3037] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:01.327143 17090 raft_consensus.cc:2454] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fcef08ad72a45a7a3f1eab2dffc2450 in term 1.
I20260923 23:07:01.333185 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.329678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.5.74.1:41527)
0923 23:07:01.329883 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:01.329885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:01.329892 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:07:01.329926 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:01.329927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:01.329948 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:01.330009 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:07:01.330189 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.330933 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.332025 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:01.332349 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:01.332409 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:01.332614 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:07:01.332758 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:07:01.332841 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:01.333078 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 23:07:01.333099 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260923 23:07:01.335315 16957 raft_consensus.cc:1261] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [term 1 FOLLOWER]: Refusing update from remote peer 0fcef08ad72a45a7a3f1eab2dffc2450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:07:01.335681 17161 consensus_queue.cc:1035] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b2a00bca5d046e3a3b2b20fc0d48e40" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:07:01.336244 17090 raft_consensus.cc:1261] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [term 1 FOLLOWER]: Refusing update from remote peer 0fcef08ad72a45a7a3f1eab2dffc2450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:07:01.338169 17167 consensus_queue.cc:1035] T 6dd9d399e6e247e0b97e634cc225bb93 P 0fcef08ad72a45a7a3f1eab2dffc2450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb9a23482f94715afd82e59ee492420" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:07:01.342301 17170 mvcc.cc:205] Tried to move back new op lower bound from 7332678948186349568 to 7332678948131430400. Current Snapshot: MvccSnapshot[applied={T|T < 7332678948186349568}]
I20260923 23:07:01.342510 17173 mvcc.cc:205] Tried to move back new op lower bound from 7332678948186349568 to 7332678948131430400. Current Snapshot: MvccSnapshot[applied={T|T < 7332678948186349568}]
I20260923 23:07:01.344655 17171 mvcc.cc:205] Tried to move back new op lower bound from 7332678948186349568 to 7332678948131430400. Current Snapshot: MvccSnapshot[applied={T|T < 7332678948186349568}]
I20260923 23:07:01.347067 16653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:07:01.347177 16653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:07:01.347862 16653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46e1d1095e8e4103a4fe53ab7b6375ed: Sending DeleteTablet for 3 replicas of tablet 6dd9d399e6e247e0b97e634cc225bb93
I20260923 23:07:01.348335 17071 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dd9d399e6e247e0b97e634cc225bb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:01 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:48596
I20260923 23:07:01.348574 16799 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dd9d399e6e247e0b97e634cc225bb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:01 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:60234
I20260923 23:07:01.348958 16937 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dd9d399e6e247e0b97e634cc225bb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:01 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:46206
I20260923 23:07:01.349448 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 16734
I20260923 23:07:01.354589 17181 tablet_replica.cc:330] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: stopping tablet replica
I20260923 23:07:01.354773 17181 raft_consensus.cc:2229] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:07:01.354864 17181 raft_consensus.cc:2258] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:07:01.355633 17179 tablet_replica.cc:330] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: stopping tablet replica
I20260923 23:07:01.355769 17179 raft_consensus.cc:2229] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:07:01.355892 17179 raft_consensus.cc:2258] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:07:01.356760 17181 ts_tablet_manager.cc:1905] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:07:01.358790 17181 ts_tablet_manager.cc:1918] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:07:01.358886 17181 log.cc:1196] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/6dd9d399e6e247e0b97e634cc225bb93
I20260923 23:07:01.359153 17181 ts_tablet_manager.cc:1939] T 6dd9d399e6e247e0b97e634cc225bb93 P 3b2a00bca5d046e3a3b2b20fc0d48e40: Deleting consensus metadata
I20260923 23:07:01.359831 17179 ts_tablet_manager.cc:1905] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:07:01.365401 17179 ts_tablet_manager.cc:1918] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:07:01.365499 17179 log.cc:1196] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/6dd9d399e6e247e0b97e634cc225bb93
I20260923 23:07:01.366187 17179 ts_tablet_manager.cc:1939] T 6dd9d399e6e247e0b97e634cc225bb93 P efb9a23482f94715afd82e59ee492420: Deleting consensus metadata
I20260923 23:07:01.366714 16640 catalog_manager.cc:4841] TS efb9a23482f94715afd82e59ee492420 (127.5.74.3:44221): tablet 6dd9d399e6e247e0b97e634cc225bb93 (table test-table [id=e5e9dcde80ee4252b47333ae95f845a4]) successfully deleted
I20260923 23:07:01.367235 16638 catalog_manager.cc:4841] TS 3b2a00bca5d046e3a3b2b20fc0d48e40 (127.5.74.2:44601): tablet 6dd9d399e6e247e0b97e634cc225bb93 (table test-table [id=e5e9dcde80ee4252b47333ae95f845a4]) successfully deleted
W20260923 23:07:01.367830 16639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:41527 (error 108)
I20260923 23:07:01.368841 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:01.368728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:41527 (local address 127.0.0.1:60258)
0923 23:07:01.368777 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:01.368807 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:41527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 23:07:01.369055 16639 catalog_manager.cc:4584] TS 0fcef08ad72a45a7a3f1eab2dffc2450 (127.5.74.1:41527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6dd9d399e6e247e0b97e634cc225bb93: Network error: Client connection negotiation failed: client connection to 127.5.74.1:41527: connect: Connection refused (error 111)
I20260923 23:07:01.369297 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 16871
I20260923 23:07:01.377072 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 17008
I20260923 23:07:01.383092 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 16621
I20260923 23:07:01.388893 5416 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 23:07:01.724774 5416 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 23:07:01.724915 5416 mini_ranger.cc:66] Stopping Ranger...
I20260923 23:07:02.070297 5416 mini_ranger.cc:68] Stopped Ranger
2026-09-23 23:07:02.070 UTC [16207] LOG: received smart shutdown request
2026-09-23 23:07:02.076 UTC [16207] LOG: background worker "logical replication launcher" (PID 16216) exited with exit code 1
2026-09-23 23:07:02.077 UTC [16211] LOG: shutting down
2026-09-23 23:07:02.383 UTC [16207] LOG: database system is shut down
2026-09-23T23:07:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50048 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:07:02 dist-test-slave-j4c5 krb5kdc[17191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:07:02 dist-test-slave-j4c5 krb5kdc[17191](info): set up 2 sockets
Sep 23 23:07:02 dist-test-slave-j4c5 krb5kdc[17191](info): commencing operation
krb5kdc: starting...
W20260923 23:07:03.560226 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:07:03 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:07:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.74.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.74.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/rangeradmin_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/rangeradmin_127.5.74.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.74.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.74.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/rangerlookup_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/rangerlookup_127.5.74.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.74.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.74.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/HTTP_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/HTTP_127.5.74.20@KRBTEST.COM.keytab.
I20260923 23:07:03.748907 5416 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcA05g1/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/postgres -l logfile start
2026-09-23 23:07:06.633 UTC [17221] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 23:07:06.634 UTC [17221] LOG: listening on IPv4 address "127.5.74.20", port 43361
2026-09-23 23:07:06.638 UTC [17221] LOG: listening on Unix socket "/tmp/.s.PGSQL.43361"
2026-09-23 23:07:06.661 UTC [17224] LOG: database system was shut down at 2026-09-23 23:07:04 UTC
2026-09-23 23:07:06.665 UTC [17221] LOG: database system is ready to accept connections
I20260923 23:07:07.632107 5416 mini_postgres.cc:96] Postgres bound to 43361
2026-09-23 23:07:07.639 UTC [17232] FATAL: database "slave" does not exist
127.5.74.20:43361 - accepting connections
I20260923 23:07:07.640038 5416 mini_ranger.cc:161] Starting Ranger...
I20260923 23:07:07.656060 5416 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 23:07:08.251116 5416 mini_ranger.cc:87] Created ranger Postgres database
I20260923 23:07:08.251230 5416 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-admin
2026-09-23 23:07:08,551 [I] DB FLAVOR :POSTGRES
2026-09-23 23:07:08,552 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 23:07:08,552 [I] Checking connection..
2026-09-23 23:07:08,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 23:07:08,828 [I] Checking connection passed.
2026-09-23 23:07:08,829 [I] --------- Verifying version history table ---------
2026-09-23 23:07:08,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:07:09,108 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 23:07:09,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:07:09,380 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 23:07:09,380 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 23:07:09,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 23:07:09.629 UTC [17296] WARNING: there is no transaction in progress
2026-09-23 23:07:09,636 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 23:07:09,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:07:09,882 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 23:07:09,883 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 23:07:09,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 23:07:10,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 23:07:10,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j4c5.c.gcp-upstream.internal','N') ;"
2026-09-23 23:07:10,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 23:07:10,927 [I] Table x_portal_user does not exist in database ranger
2026-09-23 23:07:10,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 23:07:11,180 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 23:07:11,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 23:07:11,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 23:07:11,696 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 23:07:11,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 23:07:12.843 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:12.865 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:12.887 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:12.906 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:12.924 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:12.944 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.010 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.019 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.037 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.049 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.063 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.079 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.090 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.099 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.108 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13.117 UTC [17404] WARNING: there is no transaction in progress
2026-09-23 23:07:13,952 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 23:07:13,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 23:07:14,189 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 23:07:14,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 23:07:14,469 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 23:07:14,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 23:07:14,732 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 23:07:14,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 23:07:14,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 23:07:15,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j4c5.c.gcp-upstream.internal';"
2026-09-23 23:07:15,500 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 23:07:15,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 23:07:15,740 [I] DB_PATCHES have already been applied
I20260923 23:07:15.745759 5416 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcA05g1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 23:07:15.745837 5416 mini_ranger.cc:193] Using host: 127.5.74.20
I20260923 23:07:15.749207 5416 mini_ranger.cc:227] Ranger admin URL: http://127.5.74.20:41665
Sep 23, 2026 11:07:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 11:07:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 11:07:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 11:07:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 11:07:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 11:07:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 11:07:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 23:07:16.866266 5416 mini_ranger.cc:160] Time spent starting Ranger: real 9.226s user 0.003s sys 0.006s
Sep 23, 2026 11:07:37 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 11:07:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 11:07:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 11:07:37 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 11:07:38 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 23:07:39.072742 5416 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.74.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.74.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/rangerkms_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.74.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/rangerkms_127.5.74.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.74.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.74.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.74.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/HTTP_127.5.74.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.74.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/HTTP_127.5.74.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 23:07:39 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204859, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 23:07:39.167378 5416 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 23:07:39.177625 5416 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 23:07:40.286265 5416 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 23:07:40.286381 5416 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-kms
I20260923 23:07:40.286402 5416 mini_ranger_kms.cc:227] Using postgres at 127.5.74.20:43361
2026-09-23 23:07:40,464 [I] DB FLAVOR :POSTGRES
2026-09-23 23:07:40,464 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 23:07:40,464 [I] Checking connection
2026-09-23 23:07:40,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 23:07:40,728 [I] connection success
2026-09-23 23:07:40,728 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 23:07:40,729 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 23:07:40,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 23:07:40,982 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 23:07:40,983 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 23:07:40,983 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 23:07:40,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.74.20:43361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 23:07:41,290 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 23:07:41.803072 5416 mini_ranger_kms.cc:310] Created kms service
I20260923 23:07:42.021027 5416 mini_ranger_kms.cc:326] Created kudu user
I20260923 23:07:42.073096 5416 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 23:07:42 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204859, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:42.514998 5416 mini_ranger_kms.cc:384] Added ranger policy
I20260923 23:07:42.515237 5416 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcA05g1/build/release/bin/postgresql.jar:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcA05g1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 23:07:42.515264 5416 mini_ranger_kms.cc:240] Using host: 127.5.74.20
I20260923 23:07:42.519025 5416 mini_ranger_kms.cc:276] Ranger KMS PID: 17585
I20260923 23:07:42.519109 5416 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.74.20:48109
Sep 23, 2026 11:07:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 11:07:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 11:07:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 11:07:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 11:07:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 11:07:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 23:07:44.894318 5416 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.727s user 0.005s sys 0.154s
I20260923 23:07:44.894433 5416 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 23:07:44.894485 5416 mini_ranger_kms.cc:401] 127.5.74.20:48109/kms/v1/keys
Sep 23 23:07:45 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790204865, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.74.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 23:07:45 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204865, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.74.20@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
Sep 23 23:07:45 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204865, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.74.20@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
Sep 23 23:07:48 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:07:48.304780 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41629
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:41629
--ranger_config_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:48109/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:48.420831 17635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:48.421129 17635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:48.421201 17635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:48.422614 17635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:07:48.422681 17635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:48.422698 17635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:48.422710 17635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:07:48.422722 17635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:07:48.425072 17635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.74.20:48109/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:41629
--ranger_config_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:48.425563 17635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:48.425874 17635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:48.432439 17640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:48.432443 17641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:48.432548 17635 server_base.cc:1034] running on GCE node
W20260923 23:07:48.432696 17643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:48.433108 17635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:48.433454 17635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:48.434690 17635 hybrid_clock.cc:648] HybridClock initialized: now 1790204868434666 us; error 39 us; skew 500 ppm
Sep 23 23:07:48 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204868, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:48.442646 17635 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:07:48.443184 17635 webserver.cc:460] Webserver started at http://127.5.74.62:41699/ using document root <none> and password file <none>
I20260923 23:07:48.443432 17635 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:48.443497 17635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:48.443661 17635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:07:48 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204868, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:48.506127 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "a3e1f6726b8545d687f97edef8c5a776"
format_stamp: "Formatted at 2026-09-23 23:07:48 on dist-test-slave-j4c5"
server_key: "d12ad4f909874bce340084202ba6ea5a"
server_key_iv: "cdc50f0a1fd73b153d80bf4b6971afed"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:48.506539 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "a3e1f6726b8545d687f97edef8c5a776"
format_stamp: "Formatted at 2026-09-23 23:07:48 on dist-test-slave-j4c5"
server_key: "d12ad4f909874bce340084202ba6ea5a"
server_key_iv: "cdc50f0a1fd73b153d80bf4b6971afed"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:48.508014 17635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:07:48.525265 17650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:48.525446 17635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:48.525624 17635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "a3e1f6726b8545d687f97edef8c5a776"
format_stamp: "Formatted at 2026-09-23 23:07:48 on dist-test-slave-j4c5"
server_key: "d12ad4f909874bce340084202ba6ea5a"
server_key_iv: "cdc50f0a1fd73b153d80bf4b6971afed"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:48.525767 17635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:48.551407 17635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:48.553972 17635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:48.554144 17635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:48.559265 17635 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:41629
I20260923 23:07:48.559341 17702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:41629 every 8 connection(s)
I20260923 23:07:48.559868 17635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:07:48.560704 17703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:48.563122 17703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Bootstrap starting.
I20260923 23:07:48.563761 17703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:48.564073 17703 log.cc:824] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:48.564780 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 17635
I20260923 23:07:48.564893 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:07:48.564946 17703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: No bootstrap required, opened a new log
I20260923 23:07:48.565121 5416 external_mini_cluster.cc:1428] Setting key fb00fed323ad61e41e2aae0a018cc070
I20260923 23:07:48.568207 17703 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } }
I20260923 23:07:48.568392 17703 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:48.568431 17703 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e1f6726b8545d687f97edef8c5a776, State: Initialized, Role: FOLLOWER
I20260923 23:07:48.568588 17703 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } }
I20260923 23:07:48.568661 17703 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:07:48.568691 17703 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:07:48.568735 17703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:48.569463 17703 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } }
I20260923 23:07:48.569589 17703 leader_election.cc:302] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3e1f6726b8545d687f97edef8c5a776; no voters:
I20260923 23:07:48.569785 17703 leader_election.cc:288] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:07:48.569809 17708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:07:48.570050 17708 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 1 LEADER]: Becoming Leader. State: Replica: a3e1f6726b8545d687f97edef8c5a776, State: Running, Role: LEADER
I20260923 23:07:48.570194 17708 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } }
I20260923 23:07:48.570073 17703 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:07:48.570600 17710 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3e1f6726b8545d687f97edef8c5a776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } } }
I20260923 23:07:48.570675 17710 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:48.570814 17709 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e1f6726b8545d687f97edef8c5a776. Latest consensus state: current_term: 1 leader_uuid: "a3e1f6726b8545d687f97edef8c5a776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } } }
I20260923 23:07:48.570878 17709 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:48.571692 17712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:07:48.572294 17712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:07:48.572696 17703 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 23 23:07:48 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:48.574681 17712 catalog_manager.cc:1347] Generated new cluster ID: 94827b3c9c9e44a7bfe4947a1797685e
I20260923 23:07:48.574733 17712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:07:48.579641 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:48.565752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.5.74.62:41629)
0923 23:07:48.565911 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:48.565920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:48.565954 (+ 34us) server_negotiation.cc:407] Connection header received
0923 23:07:48.566688 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:48.566697 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:48.566773 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:48.566940 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:48.567788 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:48.569202 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:48.570361 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:48.571174 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:48.571264 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:48.574601 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:48.574606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:48.574606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:48.574619 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:48.576962 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:48.577509 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:48.577513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:48.577513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:48.577624 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:48.577846 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:48.577847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:48.577847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:48.578170 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:48.578964 (+ 794us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:48.579254 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 23:07:48.579405 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260923 23:07:48.592206 17712 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:07:48.593405 17712 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:07:48.602072 17712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Generated new TSK 0
I20260923 23:07:48.602283 17712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:07:49.346021 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:48109/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41629
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:49.456135 17748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:49.456524 17748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:49.456571 17748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:49.457806 17748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:49.457877 17748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:49.457921 17748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:07:49.459712 17748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.74.20:48109/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:49.459965 17748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:49.460299 17748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:49.467680 17754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:49.467630 17756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:49.467612 17753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:49.467976 17748 server_base.cc:1034] running on GCE node
I20260923 23:07:49.468278 17748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:49.468575 17748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:49.469753 17748 hybrid_clock.cc:648] HybridClock initialized: now 1790204869469736 us; error 31 us; skew 500 ppm
Sep 23 23:07:49 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:49.473958 17748 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:07:49.474525 17748 webserver.cc:460] Webserver started at http://127.5.74.1:36153/ using document root <none> and password file <none>
I20260923 23:07:49.474830 17748 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:49.474896 17748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:49.475073 17748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:07:49 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:49.490342 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "0fc130afa1e34611aa5e1f7a2d83945d"
format_stamp: "Formatted at 2026-09-23 23:07:49 on dist-test-slave-j4c5"
server_key: "54fbd2366a53097e9fd417102e752366"
server_key_iv: "e88bd72b079c23963c7e1c77b5480d9a"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:49.490851 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "0fc130afa1e34611aa5e1f7a2d83945d"
format_stamp: "Formatted at 2026-09-23 23:07:49 on dist-test-slave-j4c5"
server_key: "54fbd2366a53097e9fd417102e752366"
server_key_iv: "e88bd72b079c23963c7e1c77b5480d9a"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:49.498588 17748 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20260923 23:07:49.518087 17765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:49.519964 17748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 23:07:49.520144 17748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "0fc130afa1e34611aa5e1f7a2d83945d"
format_stamp: "Formatted at 2026-09-23 23:07:49 on dist-test-slave-j4c5"
server_key: "54fbd2366a53097e9fd417102e752366"
server_key_iv: "e88bd72b079c23963c7e1c77b5480d9a"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:49.520328 17748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:49.540758 17748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:49.542958 17748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:49.543128 17748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:49.543398 17748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:49.543769 17748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:49.543815 17748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:49.543852 17748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:49.543879 17748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:49.551012 17748 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:46557
I20260923 23:07:49.551379 17748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:07:49.552023 17878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:46557 every 8 connection(s)
I20260923 23:07:49.554061 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 17748
I20260923 23:07:49.554148 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:07:49.554339 5416 external_mini_cluster.cc:1428] Setting key 7ed1f81c40792354b5fe3d3a045f094c
Sep 23 23:07:49 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:49.563408 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:49.552284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:50639 (local address 127.5.74.62:41629)
0923 23:07:49.552432 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:49.552435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:49.552609 (+ 174us) server_negotiation.cc:407] Connection header received
0923 23:07:49.553147 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:49.553149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:49.553165 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:49.553230 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:49.553827 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:49.554639 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:49.556143 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:49.556478 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:49.556563 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:49.559480 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:49.559482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:49.559482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:49.559492 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:49.561532 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:49.562121 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:49.562123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:49.562123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:49.562195 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:49.562527 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:49.562528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:49.562528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:49.562680 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:49.562807 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:49.563169 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 23:07:49.563303 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260923 23:07:49.563622 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:49.552340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41629 (local address 127.5.74.1:50639)
0923 23:07:49.552494 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:49.552534 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:49.553034 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:49.553299 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:49.553307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:49.553509 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:49.553720 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:49.553724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:49.554746 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:49.554747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:49.556041 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:49.556044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:49.556657 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:49.556658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:49.556821 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:49.557480 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:49.557491 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:49.559340 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:49.561650 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:49.561657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:49.561660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:49.562026 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:49.562286 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:49.562287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:49.562287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:49.562439 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:49.562797 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:49.562803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:49.562998 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 23:07:49.563129 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 23:07:49.563307 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260923 23:07:49.564146 17879 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41629
I20260923 23:07:49.564296 17879 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:49.564491 17879 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:07:49.565066 17667 ts_manager.cc:194] Registered new tserver with Master: 0fc130afa1e34611aa5e1f7a2d83945d (127.5.74.1:46557)
I20260923 23:07:49.565686 17667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:50639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:07:49.589241 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:48109/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41629
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:07:49 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 23:07:49.713088 17885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:49.713405 17885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:49.713454 17885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:49.715281 17885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:49.715344 17885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:49.715389 17885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:07:49.717764 17885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.74.20:48109/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:49.718043 17885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:49.718333 17885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:49.725601 17895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:49.725890 17892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:49.725971 17893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:49.726229 17885 server_base.cc:1034] running on GCE node
I20260923 23:07:49.726475 17885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:49.726764 17885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:49.727875 17885 hybrid_clock.cc:648] HybridClock initialized: now 1790204869727850 us; error 40 us; skew 500 ppm
Sep 23 23:07:49 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:49.731109 17885 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:07:49.731678 17885 webserver.cc:460] Webserver started at http://127.5.74.2:44273/ using document root <none> and password file <none>
I20260923 23:07:49.731947 17885 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:49.732017 17885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:49.732180 17885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:07:49 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:49.752537 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "87c9a1b65bbe46b3997f9125a4cb8590"
format_stamp: "Formatted at 2026-09-23 23:07:49 on dist-test-slave-j4c5"
server_key: "5b0b8452a2dc1a56dc6fd07797007bd4"
server_key_iv: "77ed5d0ca71ca3841834f56763ffe975"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:49.753003 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "87c9a1b65bbe46b3997f9125a4cb8590"
format_stamp: "Formatted at 2026-09-23 23:07:49 on dist-test-slave-j4c5"
server_key: "5b0b8452a2dc1a56dc6fd07797007bd4"
server_key_iv: "77ed5d0ca71ca3841834f56763ffe975"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:49.756721 17885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 23:07:49.776121 17903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:49.779467 17885 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260923 23:07:49.779626 17885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "87c9a1b65bbe46b3997f9125a4cb8590"
format_stamp: "Formatted at 2026-09-23 23:07:49 on dist-test-slave-j4c5"
server_key: "5b0b8452a2dc1a56dc6fd07797007bd4"
server_key_iv: "77ed5d0ca71ca3841834f56763ffe975"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:49.779759 17885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:49.797396 17885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:49.799532 17885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:49.799702 17885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:49.799939 17885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:49.800339 17885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:49.800395 17885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:49.800442 17885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:49.800472 17885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:49.807044 17885 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:35519
I20260923 23:07:49.807487 17885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:07:49.812376 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 17885
I20260923 23:07:49.812518 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:07:49.812726 5416 external_mini_cluster.cc:1428] Setting key 7121ae7888f6307cf645fa5dbd2a51fe
I20260923 23:07:49.808990 18016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:35519 every 8 connection(s)
Sep 23 23:07:49 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:49.821614 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:49.810097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:58727 (local address 127.5.74.62:41629)
0923 23:07:49.810148 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:49.810167 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:49.810740 (+ 573us) server_negotiation.cc:407] Connection header received
0923 23:07:49.811480 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:49.811482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:49.811494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:49.811525 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:49.812459 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:49.813204 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:49.814886 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:49.815219 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:49.815264 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:49.818048 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:49.818051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:49.818051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:49.818059 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:49.819810 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:49.820364 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:49.820366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:49.820366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:49.820425 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:49.820720 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:49.820721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:49.820721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:49.820861 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:49.820914 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:49.821284 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 23:07:49.821509 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:07:49.822217 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:49.809805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41629 (local address 127.5.74.2:58727)
0923 23:07:49.810583 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:49.810628 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:49.811339 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:49.811709 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:49.811714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:49.811973 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:49.812325 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:49.812332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:49.813311 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:49.813312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:49.814784 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:49.814787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:49.815384 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:49.815385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:49.815528 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:49.816230 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:49.816240 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:49.817932 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:49.819907 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:49.819914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:49.819917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:49.820261 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:49.820499 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:49.820499 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:49.820499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:49.820645 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:49.820947 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:49.820953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:49.821135 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 23:07:49.821789 (+ 654us) client_negotiation.cc:240] Negotiation successful
0923 23:07:49.821969 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":85,"threads_started":1}
I20260923 23:07:49.822690 18017 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41629
I20260923 23:07:49.822794 18017 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:49.822993 18017 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
I20260923 23:07:49.823342 17667 ts_manager.cc:194] Registered new tserver with Master: 87c9a1b65bbe46b3997f9125a4cb8590 (127.5.74.2:35519)
I20260923 23:07:49.823884 17667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:58727
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:07:49.863003 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:48109/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41629
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:49.978295 18022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:49.978514 18022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:49.978538 18022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:49.979737 18022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:49.979782 18022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:49.979810 18022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:07:49.981413 18022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.74.20:48109/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:41629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:49.981662 18022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:49.981928 18022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:49.988447 18027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:49.994959 18022 server_base.cc:1034] running on GCE node
W20260923 23:07:49.995184 18028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:49.995715 18030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:49.996024 18022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:49.996362 18022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:49.997552 18022 hybrid_clock.cc:648] HybridClock initialized: now 1790204869997521 us; error 48 us; skew 500 ppm
Sep 23 23:07:49 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:50.001201 18022 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:07:50.001749 18022 webserver.cc:460] Webserver started at http://127.5.74.3:35167/ using document root <none> and password file <none>
I20260923 23:07:50.002028 18022 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:50.002079 18022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:50.002204 18022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:50.026206 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "1bd2776de09e49a58a01941cd3d79ca6"
format_stamp: "Formatted at 2026-09-23 23:07:50 on dist-test-slave-j4c5"
server_key: "0f7cb7c13ef7280e14de3d053f8024ae"
server_key_iv: "cc20ce6d0ee7a57a0ab5d29ecfef4c6c"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:50.026662 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "1bd2776de09e49a58a01941cd3d79ca6"
format_stamp: "Formatted at 2026-09-23 23:07:50 on dist-test-slave-j4c5"
server_key: "0f7cb7c13ef7280e14de3d053f8024ae"
server_key_iv: "cc20ce6d0ee7a57a0ab5d29ecfef4c6c"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:50.028940 18022 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 23:07:50.046536 18037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.048348 18022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 23:07:50.048481 18022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "1bd2776de09e49a58a01941cd3d79ca6"
format_stamp: "Formatted at 2026-09-23 23:07:50 on dist-test-slave-j4c5"
server_key: "0f7cb7c13ef7280e14de3d053f8024ae"
server_key_iv: "cc20ce6d0ee7a57a0ab5d29ecfef4c6c"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:50.048588 18022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:50.069691 18022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:50.072306 18022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:50.072476 18022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:50.072721 18022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:50.073078 18022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:50.073130 18022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.073168 18022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:50.073185 18022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.081467 18022 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:44555
I20260923 23:07:50.082078 18022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:07:50.082636 18150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:44555 every 8 connection(s)
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:50.092329 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 18022
I20260923 23:07:50.092448 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:07:50.092662 5416 external_mini_cluster.cc:1428] Setting key 25569deb14dd02243ef4172f15aa0e84
I20260923 23:07:50.095350 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:50.083244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:33723 (local address 127.5.74.62:41629)
0923 23:07:50.083289 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:50.083291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:50.083303 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:07:50.083930 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:50.083932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:50.083948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:50.084035 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:50.084792 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.085581 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.087096 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.087420 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.087470 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:50.090265 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:50.090267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:50.090267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:50.090276 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:50.092065 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.094142 (+ 2077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.094144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.094144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.094207 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.094527 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.094528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.094529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.094677 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:50.094730 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:50.095144 (+ 414us) server_negotiation.cc:299] Negotiation successful
0923 23:07:50.095245 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:07:50.096220 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:50.082892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41629 (local address 127.5.74.3:33723)
0923 23:07:50.083087 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:50.083135 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:50.083779 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:50.084112 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:50.084116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:50.084397 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:50.084684 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:50.084690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.085698 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.085699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:50.086990 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:50.086993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.087572 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.087573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:50.087719 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:50.088455 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:50.088468 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:50.090137 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:50.093644 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:50.093654 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:50.093657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:50.094043 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:50.094292 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:50.094293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:50.094293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:50.094443 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:50.094775 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:50.094781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:50.094971 (+ 190us) client_negotiation.cc:769] Sending connection context
0923 23:07:50.095722 (+ 751us) client_negotiation.cc:240] Negotiation successful
0923 23:07:50.095922 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260923 23:07:50.097085 18151 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41629
I20260923 23:07:50.097249 18151 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.097500 18151 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
I20260923 23:07:50.097970 17667 ts_manager.cc:194] Registered new tserver with Master: 1bd2776de09e49a58a01941cd3d79ca6 (127.5.74.3:44555)
I20260923 23:07:50.098512 17667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:33723
I20260923 23:07:50.104664 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:07:50.105346 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 17748
I20260923 23:07:50.112114 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 17885
I20260923 23:07:50.116868 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 18022
I20260923 23:07:50.121865 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 17635
I20260923 23:07:50.126657 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41629
--webserver_interface=127.5.74.62
--webserver_port=41699
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:41629
--ranger_config_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:48109/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.74.20:48109/kms with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:50.209367 18154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:50.209686 18154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:50.209724 18154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:50.210915 18154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:07:50.210991 18154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:50.211015 18154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:50.211035 18154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:07:50.211066 18154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:07:50.212699 18154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.74.20:48109/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:41629
--ranger_config_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=41699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:50.212989 18154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:50.213244 18154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:50.218217 18159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:50.218217 18160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:50.218236 18162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:50.218593 18154 server_base.cc:1034] running on GCE node
I20260923 23:07:50.218874 18154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:50.219185 18154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:50.220355 18154 hybrid_clock.cc:648] HybridClock initialized: now 1790204870220349 us; error 39 us; skew 500 ppm
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:50.222977 18154 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:07:50.223390 18154 webserver.cc:460] Webserver started at http://127.5.74.62:41699/ using document root <none> and password file <none>
I20260923 23:07:50.223634 18154 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:50.223685 18154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:50.283198 18154 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:50.283936 18170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.284142 18154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:50.284261 18154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "a3e1f6726b8545d687f97edef8c5a776"
format_stamp: "Formatted at 2026-09-23 23:07:48 on dist-test-slave-j4c5"
server_key: "d12ad4f909874bce340084202ba6ea5a"
server_key_iv: "cdc50f0a1fd73b153d80bf4b6971afed"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:50.284633 18154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:50.304983 18154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:50.307125 18154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:50.307247 18154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:50.311878 18154 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:41629
I20260923 23:07:50.311939 18222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:41629 every 8 connection(s)
I20260923 23:07:50.312328 18154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:07:50.313396 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 18154
I20260923 23:07:50.313688 18223 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:07:50.313709 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:48109/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:46557
--local_ip_for_outbound_sockets=127.5.74.1
--tserver_master_addrs=127.5.74.62:41629
--webserver_port=36153
--webserver_interface=127.5.74.1
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.74.20:48109/kms with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:07:50.314416 18223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Bootstrap starting.
I20260923 23:07:50.322569 18223 log.cc:824] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:50.324677 18223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:07:50.324985 18223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Bootstrap complete.
I20260923 23:07:50.327389 18223 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } }
I20260923 23:07:50.327657 18223 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e1f6726b8545d687f97edef8c5a776, State: Initialized, Role: FOLLOWER
I20260923 23:07:50.327847 18223 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } }
I20260923 23:07:50.327925 18223 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:07:50.327980 18223 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:07:50.328028 18223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:07:50.329078 18223 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } }
I20260923 23:07:50.329200 18223 leader_election.cc:302] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3e1f6726b8545d687f97edef8c5a776; no voters:
I20260923 23:07:50.329419 18223 leader_election.cc:288] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 23:07:50.329452 18227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:07:50.329562 18227 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [term 2 LEADER]: Becoming Leader. State: Replica: a3e1f6726b8545d687f97edef8c5a776, State: Running, Role: LEADER
I20260923 23:07:50.329694 18223 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:07:50.329725 18227 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } }
I20260923 23:07:50.330090 18229 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e1f6726b8545d687f97edef8c5a776. Latest consensus state: current_term: 2 leader_uuid: "a3e1f6726b8545d687f97edef8c5a776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } } }
I20260923 23:07:50.330224 18228 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a3e1f6726b8545d687f97edef8c5a776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e1f6726b8545d687f97edef8c5a776" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41629 } } }
I20260923 23:07:50.330417 18229 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:50.330428 18228 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:50.330649 18236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:07:50.331244 18236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:07:50.331429 18236 catalog_manager.cc:1259] Loaded cluster ID: 94827b3c9c9e44a7bfe4947a1797685e
I20260923 23:07:50.331569 18236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:07:50.331679 18223 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 23:07:50.332860 18236 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:07:50.333050 18236 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Loaded TSK: 0
I20260923 23:07:50.333312 18236 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:07:50.427616 18225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:50.427857 18225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:50.427898 18225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:50.429566 18225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:50.429641 18225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:50.429697 18225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:07:50.431598 18225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.74.20:48109/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:46557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=36153
--tserver_master_addrs=127.5.74.62:41629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:50.431900 18225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:50.432195 18225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:50.437206 18261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:50.437223 18263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:50.437705 18260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:50.437955 18225 server_base.cc:1034] running on GCE node
I20260923 23:07:50.438158 18225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:50.438438 18225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:50.441700 18225 hybrid_clock.cc:648] HybridClock initialized: now 1790204870441659 us; error 50 us; skew 500 ppm
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:50.444803 18225 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:07:50.445263 18225 webserver.cc:460] Webserver started at http://127.5.74.1:36153/ using document root <none> and password file <none>
I20260923 23:07:50.445554 18225 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:50.445652 18225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:50.472661 18225 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260923 23:07:50.473851 18271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.474139 18225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:50.474252 18225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "0fc130afa1e34611aa5e1f7a2d83945d"
format_stamp: "Formatted at 2026-09-23 23:07:49 on dist-test-slave-j4c5"
server_key: "54fbd2366a53097e9fd417102e752366"
server_key_iv: "e88bd72b079c23963c7e1c77b5480d9a"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:50.474773 18225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:50.494354 18225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:50.496991 18225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:50.497157 18225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:50.497442 18225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:50.497824 18225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:50.497913 18225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.497990 18225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:50.498083 18225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.505568 18225 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:46557
I20260923 23:07:50.505666 18384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:46557 every 8 connection(s)
I20260923 23:07:50.506143 18225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:50.515220 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 18225
I20260923 23:07:50.515555 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:48109/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:35519
--local_ip_for_outbound_sockets=127.5.74.2
--tserver_master_addrs=127.5.74.62:41629
--webserver_port=44273
--webserver_interface=127.5.74.2
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.74.20:48109/kms with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:07:50.518396 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:50.507044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:43093 (local address 127.5.74.62:41629)
0923 23:07:50.507220 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:50.507233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:50.507263 (+ 30us) server_negotiation.cc:407] Connection header received
0923 23:07:50.508118 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:50.508125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:50.508206 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:50.508317 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:50.509030 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.509908 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.511171 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.511804 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.511933 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:50.514316 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:50.514321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:50.514322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:50.514335 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:50.516461 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.516995 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.516998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.516999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.517112 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.517385 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.517386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.517387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.517650 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:50.517746 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:50.518059 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 23:07:50.518171 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260923 23:07:50.518419 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:50.506871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41629 (local address 127.5.74.1:43093)
0923 23:07:50.507069 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:50.507117 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:50.507849 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:50.508392 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:50.508397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:50.508628 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:50.508911 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:50.508917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.510051 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.510052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:50.511083 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:50.511086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.511941 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.511942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:50.512041 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:50.512680 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:50.512688 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:50.514194 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:50.516564 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:50.516572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:50.516574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:50.516898 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:50.517187 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:50.517188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:50.517188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:50.517300 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:50.517738 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:50.517744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:50.517883 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 23:07:50.517994 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 23:07:50.518171 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":121,"threads_started":1}
I20260923 23:07:50.519210 18385 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41629
I20260923 23:07:50.519313 18385 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.519546 18385 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
W20260923 23:07:50.520555 18385 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:41629 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 23:07:50.520637 18385 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.520749 18385 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
I20260923 23:07:50.521229 18385 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.521314 18385 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
W20260923 23:07:50.521726 18385 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:07:50.620708 18389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:50.620997 18389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:50.621073 18389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:50.622928 18389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:50.623054 18389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:50.623131 18389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:07:50.625612 18389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.74.20:48109/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:35519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=44273
--tserver_master_addrs=127.5.74.62:41629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:50.625900 18389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:50.626207 18389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:50.633622 18397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:50.633929 18389 server_base.cc:1034] running on GCE node
W20260923 23:07:50.634025 18395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:50.633438 18394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:50.636651 18389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:50.636945 18389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:50.638113 18389 hybrid_clock.cc:648] HybridClock initialized: now 1790204870638083 us; error 53 us; skew 500 ppm
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:50.641115 18389 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:07:50.641678 18389 webserver.cc:460] Webserver started at http://127.5.74.2:44273/ using document root <none> and password file <none>
I20260923 23:07:50.641942 18389 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:50.642024 18389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:50.662774 18389 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.001s
I20260923 23:07:50.663836 18405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.664062 18389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:07:50.664204 18389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "87c9a1b65bbe46b3997f9125a4cb8590"
format_stamp: "Formatted at 2026-09-23 23:07:49 on dist-test-slave-j4c5"
server_key: "5b0b8452a2dc1a56dc6fd07797007bd4"
server_key_iv: "77ed5d0ca71ca3841834f56763ffe975"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:50.664633 18389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:50.685252 18389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:50.687683 18389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:50.687852 18389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:50.688192 18389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:50.688650 18389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:50.688705 18389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.688737 18389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:50.688797 18389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.696480 18389 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:35519
I20260923 23:07:50.696564 18518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:35519 every 8 connection(s)
I20260923 23:07:50.697046 18389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:50.704358 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 18389
I20260923 23:07:50.704701 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.74.20:48109/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:44555
--local_ip_for_outbound_sockets=127.5.74.3
--tserver_master_addrs=127.5.74.62:41629
--webserver_port=35167
--webserver_interface=127.5.74.3
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.74.20:48109/kms with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:07:50.712704 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:50.697858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:33055 (local address 127.5.74.62:41629)
0923 23:07:50.697939 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:50.697942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:50.698133 (+ 191us) server_negotiation.cc:407] Connection header received
0923 23:07:50.698668 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:50.698670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:50.698686 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:50.698735 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:50.699297 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.699834 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.701171 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.701462 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.701547 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:50.703984 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:50.703986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:50.703987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:50.703996 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:50.705768 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.706555 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.706556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.706557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.706631 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.706953 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.706954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.706954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.707117 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:50.707215 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:50.712452 (+ 5237us) server_negotiation.cc:299] Negotiation successful
0923 23:07:50.712584 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 23:07:50.713630 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:50.697826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41629 (local address 127.5.74.2:33055)
0923 23:07:50.697990 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:50.698040 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:50.698555 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:50.698807 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:50.698816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:50.699005 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:50.699198 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:50.699202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.699942 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.699943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:50.701077 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:50.701080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.701591 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.701593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:50.701755 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:50.702331 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:50.702341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:50.703829 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:50.705842 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:50.705851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:50.705854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:50.706179 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:50.706710 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:50.706711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:50.706711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:50.706861 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:50.707265 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:50.707271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:50.707470 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 23:07:50.713120 (+ 5650us) client_negotiation.cc:240] Negotiation successful
0923 23:07:50.713351 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260923 23:07:50.714514 18519 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41629
I20260923 23:07:50.714689 18519 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.714995 18519 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
W20260923 23:07:50.715746 18519 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:41629 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 23:07:50.715875 18519 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.716004 18519 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
I20260923 23:07:50.716408 18519 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.716529 18519 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
W20260923 23:07:50.716871 18519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:07:50.824132 18522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:50.824522 18522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:50.824566 18522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:50.826540 18522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:50.826647 18522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:50.826713 18522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:07:50.829243 18522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.74.20:48109/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:44555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=35167
--tserver_master_addrs=127.5.74.62:41629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:50.829552 18522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:50.829866 18522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:50.836589 18530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:50.836757 18527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:50.836970 18528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:50.839854 18522 server_base.cc:1034] running on GCE node
I20260923 23:07:50.840114 18522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:50.840497 18522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:50.841681 18522 hybrid_clock.cc:648] HybridClock initialized: now 1790204870841657 us; error 36 us; skew 500 ppm
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:50.845371 18522 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:07:50.846225 18522 webserver.cc:460] Webserver started at http://127.5.74.3:35167/ using document root <none> and password file <none>
I20260923 23:07:50.846557 18522 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:50.846704 18522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:50.861604 18522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:50.862484 18539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.862684 18522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:07:50.862776 18522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "1bd2776de09e49a58a01941cd3d79ca6"
format_stamp: "Formatted at 2026-09-23 23:07:50 on dist-test-slave-j4c5"
server_key: "0f7cb7c13ef7280e14de3d053f8024ae"
server_key_iv: "cc20ce6d0ee7a57a0ab5d29ecfef4c6c"
server_key_version: "kuduclusterkey@0"
I20260923 23:07:50.863190 18522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:50.886837 18522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:50.889113 18522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:50.889298 18522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:50.889612 18522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:50.890025 18522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:50.890079 18522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.890120 18522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:50.890147 18522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:50.897454 18522 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:44555
I20260923 23:07:50.897536 18652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:44555 every 8 connection(s)
I20260923 23:07:50.897936 18522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:07:50.902194 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 18522
Sep 23 23:07:50 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:50.911298 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:50.898789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:54703 (local address 127.5.74.62:41629)
0923 23:07:50.898836 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:50.898838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:50.899063 (+ 225us) server_negotiation.cc:407] Connection header received
0923 23:07:50.899752 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:50.899753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:50.899768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:50.899800 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:50.900849 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.901590 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.903656 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.903949 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.904036 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:50.907779 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:50.907781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:50.907781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:50.907789 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:50.909614 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.910148 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.910150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.910150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.910215 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.910592 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.910593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.910594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.910708 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:50.910805 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:50.911116 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 23:07:50.911200 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:07:50.911533 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:50.898784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41629 (local address 127.5.74.3:54703)
0923 23:07:50.898950 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:50.898988 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:50.899620 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:50.900068 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:50.900073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:50.900414 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:50.900715 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:50.900721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.901709 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.901710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:50.903556 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:50.903560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.904070 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.904071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:50.904251 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:50.904864 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:50.904875 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:50.907655 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:50.909711 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:50.909719 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:50.909722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:50.910055 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:50.910290 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:50.910290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:50.910291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:50.910504 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:50.910796 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:50.910803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:50.910972 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 23:07:50.911093 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 23:07:50.911267 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260923 23:07:50.911975 18653 heartbeater.cc:347] Connected to a master server at 127.5.74.62:41629
I20260923 23:07:50.912079 18653 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.912348 18653 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
W20260923 23:07:50.913064 18653 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:41629 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 23:07:50.913146 18653 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.913220 18653 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
I20260923 23:07:50.913527 18653 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:50.913604 18653 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
W20260923 23:07:50.913954 18653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:07:50.916332 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:50.902744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.5.74.62:41629)
0923 23:07:50.902864 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:50.902866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:50.904942 (+ 2076us) server_negotiation.cc:407] Connection header received
0923 23:07:50.904971 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:50.904971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:50.904986 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:50.905063 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:50.907518 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.908300 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.911659 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:50.911967 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:50.912037 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:50.913538 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:50.913540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:50.913540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:50.913548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:50.915029 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.915454 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.915455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.915455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.915506 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:50.915712 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:50.915713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:50.915713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:50.915844 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:50.916032 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:50.916117 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 23:07:50.916230 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
Sep 23 23:07:51 dist-test-slave-j4c5 krb5kdc[17191](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790204871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:51.521963 18385 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:51.522141 18385 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
I20260923 23:07:51.522661 18187 ts_manager.cc:194] Registered new tserver with Master: 0fc130afa1e34611aa5e1f7a2d83945d (127.5.74.1:46557)
I20260923 23:07:51.523720 18187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:43093
I20260923 23:07:51.717118 18519 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:51.717367 18519 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
I20260923 23:07:51.717805 18187 ts_manager.cc:194] Registered new tserver with Master: 87c9a1b65bbe46b3997f9125a4cb8590 (127.5.74.2:35519)
I20260923 23:07:51.718246 18187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:33055
Sep 23 23:07:51 dist-test-slave-j4c5 krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for HTTP/127.5.74.20@KRBTEST.COM
I20260923 23:07:51.914152 18653 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:51.914381 18653 heartbeater.cc:511] Master 127.5.74.62:41629 requested a full tablet report, sending...
I20260923 23:07:51.914752 18187 ts_manager.cc:194] Registered new tserver with Master: 1bd2776de09e49a58a01941cd3d79ca6 (127.5.74.3:44555)
I20260923 23:07:51.915223 18187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:54703
I20260923 23:07:51.922034 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:07:51.929839 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.923025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.5.74.62:41629)
0923 23:07:51.923143 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:51.923145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:51.923182 (+ 37us) server_negotiation.cc:407] Connection header received
0923 23:07:51.923262 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:51.923263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:51.923275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:51.923332 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:51.923888 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.924477 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.925214 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.925445 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.925549 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.926915 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:51.926917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:51.926918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:51.926925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:51.928590 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:51.928969 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:51.928971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:51.928971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:51.929033 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:51.929244 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:51.929245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:51.929246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:51.929374 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:51.929486 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:51.929628 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 23:07:51.929721 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260923 23:07:51.932140 18187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:07:51.934072 18187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:07:51.940974 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.936778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:46557 (local address 127.0.0.1:50608)
0923 23:07:51.937401 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:51.937445 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:51.937506 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:51.937783 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:51.937786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:51.937788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:51.937870 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.937872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.938753 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.938754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.939748 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.939750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.940626 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.940627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.940707 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.940721 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:07:51.940822 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 23:07:51.940877 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":89,"threads_started":1}
I20260923 23:07:51.940975 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.937259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.5.74.1:46557)
0923 23:07:51.937640 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:51.937645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:51.937655 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:07:51.937689 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:51.937691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:51.937707 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:51.937781 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:51.937967 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.938661 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.939837 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.940526 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.940613 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.940678 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:51.940840 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 23:07:51.940884 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":45,"threads_started":1}
I20260923 23:07:51.942359 18305 tablet_service.cc:1467] Processing CreateTablet for tablet ea496383736c44b0b122139ba26de8ea (DEFAULT_TABLE table=test-table [id=76e91ba164144ec1bf333e5cc73fa70f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:07:51.942767 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea496383736c44b0b122139ba26de8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:51.945724 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.936778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35519 (local address 127.0.0.1:39686)
0923 23:07:51.941632 (+ 4854us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:51.941645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:51.941699 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:51.942008 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:51.942009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:51.942010 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:51.942041 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.942042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.943370 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.943371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.944437 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.944441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.945480 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.945481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.945543 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.945547 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:07:51.945601 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 23:07:51.945642 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4839,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260923 23:07:51.946305 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.937049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.5.74.2:35519)
0923 23:07:51.937916 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:51.937921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:51.941772 (+ 3851us) server_negotiation.cc:407] Connection header received
0923 23:07:51.941816 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:51.941818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:51.941834 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:51.941907 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:51.942110 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.943275 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.944540 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.945294 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.945367 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.945442 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:51.946174 (+ 732us) server_negotiation.cc:299] Negotiation successful
0923 23:07:51.946223 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":53,"threads_started":1}
I20260923 23:07:51.947149 18453 tablet_service.cc:1467] Processing CreateTablet for tablet ea496383736c44b0b122139ba26de8ea (DEFAULT_TABLE table=test-table [id=76e91ba164144ec1bf333e5cc73fa70f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:07:51.947520 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea496383736c44b0b122139ba26de8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:51.947968 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.937239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.5.74.3:44555)
0923 23:07:51.937702 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:51.937706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:51.944321 (+ 6615us) server_negotiation.cc:407] Connection header received
0923 23:07:51.944468 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:51.944471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:51.944488 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:51.944539 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:51.944772 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.945794 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.947055 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.947604 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.947698 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.947758 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:51.947851 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 23:07:51.947892 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":51,"threads_started":1}
I20260923 23:07:51.947966 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.936995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:44555 (local address 127.0.0.1:50576)
0923 23:07:51.944230 (+ 7235us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:51.944243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:51.944383 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:51.944653 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:51.944653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:51.944655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:51.944698 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.944699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.945910 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.945910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.946940 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.946943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.947712 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.947713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.947776 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.947780 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:07:51.947837 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 23:07:51.947877 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7215,"thread_start_us":63,"threads_started":1}
I20260923 23:07:51.948626 18587 tablet_service.cc:1467] Processing CreateTablet for tablet ea496383736c44b0b122139ba26de8ea (DEFAULT_TABLE table=test-table [id=76e91ba164144ec1bf333e5cc73fa70f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:07:51.948863 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea496383736c44b0b122139ba26de8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:51.951141 18687 tablet_bootstrap.cc:492] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590: Bootstrap starting.
I20260923 23:07:51.951820 18687 tablet_bootstrap.cc:654] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:51.952252 18688 tablet_bootstrap.cc:492] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: Bootstrap starting.
I20260923 23:07:51.952277 18687 log.cc:824] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:51.952900 18688 tablet_bootstrap.cc:654] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:51.953183 18688 log.cc:824] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:51.953886 18688 tablet_bootstrap.cc:492] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: No bootstrap required, opened a new log
I20260923 23:07:51.953974 18688 ts_tablet_manager.cc:1397] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:07:51.956584 18688 raft_consensus.cc:348] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } }
I20260923 23:07:51.956773 18688 raft_consensus.cc:374] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:51.956846 18688 raft_consensus.cc:729] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd2776de09e49a58a01941cd3d79ca6, State: Initialized, Role: FOLLOWER
I20260923 23:07:51.957003 18688 consensus_queue.cc:260] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } }
I20260923 23:07:51.957360 18688 ts_tablet_manager.cc:1428] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:07:51.957549 18653 heartbeater.cc:503] Master 127.5.74.62:41629 was elected leader, sending a full tablet report...
I20260923 23:07:51.960361 18686 tablet_bootstrap.cc:492] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d: Bootstrap starting.
I20260923 23:07:51.961010 18686 tablet_bootstrap.cc:654] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:51.961314 18686 log.cc:824] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:51.962119 18686 tablet_bootstrap.cc:492] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d: No bootstrap required, opened a new log
I20260923 23:07:51.962205 18686 ts_tablet_manager.cc:1397] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:07:51.964933 18687 tablet_bootstrap.cc:492] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590: No bootstrap required, opened a new log
I20260923 23:07:51.965014 18687 ts_tablet_manager.cc:1397] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260923 23:07:51.964926 18686 raft_consensus.cc:348] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } }
I20260923 23:07:51.965109 18686 raft_consensus.cc:374] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:51.965162 18686 raft_consensus.cc:729] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc130afa1e34611aa5e1f7a2d83945d, State: Initialized, Role: FOLLOWER
I20260923 23:07:51.965304 18686 consensus_queue.cc:260] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } }
I20260923 23:07:51.965636 18686 ts_tablet_manager.cc:1428] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:07:51.965772 18385 heartbeater.cc:503] Master 127.5.74.62:41629 was elected leader, sending a full tablet report...
I20260923 23:07:51.967711 18687 raft_consensus.cc:348] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } }
I20260923 23:07:51.967839 18687 raft_consensus.cc:374] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:51.967859 18687 raft_consensus.cc:729] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c9a1b65bbe46b3997f9125a4cb8590, State: Initialized, Role: FOLLOWER
I20260923 23:07:51.967986 18687 consensus_queue.cc:260] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } }
I20260923 23:07:51.968334 18519 heartbeater.cc:503] Master 127.5.74.62:41629 was elected leader, sending a full tablet report...
I20260923 23:07:51.968521 18691 raft_consensus.cc:482] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:07:51.968629 18691 raft_consensus.cc:504] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } }
I20260923 23:07:51.968709 18687 ts_tablet_manager.cc:1428] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 23:07:51.969007 18691 leader_election.cc:288] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c9a1b65bbe46b3997f9125a4cb8590 (127.5.74.2:35519), 0fc130afa1e34611aa5e1f7a2d83945d (127.5.74.1:46557)
I20260923 23:07:51.973682 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.969315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:60321 (local address 127.5.74.2:35519)
0923 23:07:51.969358 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:51.969361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:51.969495 (+ 134us) server_negotiation.cc:407] Connection header received
0923 23:07:51.969600 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:51.969602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:51.969613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:51.969644 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:51.970386 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.971132 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.972708 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.973262 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.973309 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.973319 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:51.973575 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 23:07:51.973614 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:07:51.973683 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.969517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:51231 (local address 127.5.74.1:46557)
0923 23:07:51.969553 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:51.969555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:51.970051 (+ 496us) server_negotiation.cc:407] Connection header received
0923 23:07:51.970150 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:51.970151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:51.970162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:51.970191 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:51.970419 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.971118 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.972708 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.973262 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.973309 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.973319 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:51.973575 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 23:07:51.973614 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:07:51.973963 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.969303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:46557 (local address 127.5.74.3:51231)
0923 23:07:51.969993 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:51.970003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:51.970077 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:51.970254 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:51.970255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:51.970257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:51.970325 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.970327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.971224 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.971225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.972602 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.972605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.973411 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.973413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.973486 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.973491 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:07:51.973804 (+ 313us) client_negotiation.cc:240] Negotiation successful
0923 23:07:51.973862 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":61,"thread_start_us":67,"threads_started":1}
I20260923 23:07:51.974323 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.969146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35519 (local address 127.5.74.3:60321)
0923 23:07:51.969412 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:51.969424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:51.969523 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:51.969714 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:51.969715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:51.969717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:51.969795 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.969797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.971224 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.971225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.972603 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:51.972605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.973421 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.973422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:51.973486 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.973490 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:07:51.974203 (+ 713us) client_negotiation.cc:240] Negotiation successful
0923 23:07:51.974247 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":72,"thread_start_us":93,"threads_started":1}
I20260923 23:07:51.974324 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea496383736c44b0b122139ba26de8ea" candidate_uuid: "1bd2776de09e49a58a01941cd3d79ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" is_pre_election: true
I20260923 23:07:51.974488 18325 raft_consensus.cc:2454] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd2776de09e49a58a01941cd3d79ca6 in term 0.
I20260923 23:07:51.974707 18542 leader_election.cc:302] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fc130afa1e34611aa5e1f7a2d83945d, 1bd2776de09e49a58a01941cd3d79ca6; no voters:
I20260923 23:07:51.974867 18691 raft_consensus.cc:2781] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:07:51.974831 18458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea496383736c44b0b122139ba26de8ea" candidate_uuid: "1bd2776de09e49a58a01941cd3d79ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" is_pre_election: true
I20260923 23:07:51.974908 18691 raft_consensus.cc:482] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:07:51.974931 18691 raft_consensus.cc:3037] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:51.974979 18458 raft_consensus.cc:2454] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd2776de09e49a58a01941cd3d79ca6 in term 0.
I20260923 23:07:51.975919 18691 raft_consensus.cc:504] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } }
I20260923 23:07:51.976073 18691 leader_election.cc:288] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 87c9a1b65bbe46b3997f9125a4cb8590 (127.5.74.2:35519), 0fc130afa1e34611aa5e1f7a2d83945d (127.5.74.1:46557)
I20260923 23:07:51.976248 18458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea496383736c44b0b122139ba26de8ea" candidate_uuid: "1bd2776de09e49a58a01941cd3d79ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c9a1b65bbe46b3997f9125a4cb8590"
I20260923 23:07:51.976315 18458 raft_consensus.cc:3037] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:51.976811 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea496383736c44b0b122139ba26de8ea" candidate_uuid: "1bd2776de09e49a58a01941cd3d79ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc130afa1e34611aa5e1f7a2d83945d"
I20260923 23:07:51.976887 18325 raft_consensus.cc:3037] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:51.977262 18458 raft_consensus.cc:2454] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd2776de09e49a58a01941cd3d79ca6 in term 1.
I20260923 23:07:51.977439 18543 leader_election.cc:302] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd2776de09e49a58a01941cd3d79ca6, 87c9a1b65bbe46b3997f9125a4cb8590; no voters:
I20260923 23:07:51.977515 18691 raft_consensus.cc:2781] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:07:51.977645 18691 raft_consensus.cc:686] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd2776de09e49a58a01941cd3d79ca6, State: Running, Role: LEADER
I20260923 23:07:51.977751 18691 consensus_queue.cc:237] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } }
I20260923 23:07:51.977831 18325 raft_consensus.cc:2454] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd2776de09e49a58a01941cd3d79ca6 in term 1.
I20260923 23:07:51.978581 18187 catalog_manager.cc:5495] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bd2776de09e49a58a01941cd3d79ca6 (127.5.74.3). New cstate: current_term: 1 leader_uuid: "1bd2776de09e49a58a01941cd3d79ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bd2776de09e49a58a01941cd3d79ca6" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 44555 } health_report { overall_health: HEALTHY } } }
I20260923 23:07:51.989109 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:51.986172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.5.74.3:44555)
0923 23:07:51.986212 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:51.986214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:51.986222 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:07:51.986305 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:51.986306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:51.986326 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:51.986411 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:07:51.986542 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.987209 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.988210 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:51.988433 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:51.988510 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:51.988675 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:07:51.988789 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:07:51.988888 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:51.988907 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 23:07:51.988928 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:07:51.991078 18458 raft_consensus.cc:1261] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590 [term 1 FOLLOWER]: Refusing update from remote peer 1bd2776de09e49a58a01941cd3d79ca6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:07:51.991078 18325 raft_consensus.cc:1261] T ea496383736c44b0b122139ba26de8ea P 0fc130afa1e34611aa5e1f7a2d83945d [term 1 FOLLOWER]: Refusing update from remote peer 1bd2776de09e49a58a01941cd3d79ca6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:07:51.991432 18697 consensus_queue.cc:1035] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fc130afa1e34611aa5e1f7a2d83945d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:07:51.991533 18691 consensus_queue.cc:1035] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c9a1b65bbe46b3997f9125a4cb8590" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:07:51.994858 18701 mvcc.cc:205] Tried to move back new op lower bound from 7332679155672731648 to 7332679155621556224. Current Snapshot: MvccSnapshot[applied={T|T < 7332679155672731648}]
I20260923 23:07:51.995924 18700 mvcc.cc:205] Tried to move back new op lower bound from 7332679155672731648 to 7332679155621556224. Current Snapshot: MvccSnapshot[applied={T|T < 7332679155672731648}]
I20260923 23:07:51.998605 18187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:07:51.998704 18187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:07:51.999570 18187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3e1f6726b8545d687f97edef8c5a776: Sending DeleteTablet for 3 replicas of tablet ea496383736c44b0b122139ba26de8ea
I20260923 23:07:51.999907 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet ea496383736c44b0b122139ba26de8ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:51 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:50576
I20260923 23:07:51.999971 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet ea496383736c44b0b122139ba26de8ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:51 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:39686
I20260923 23:07:52.000207 18305 tablet_service.cc:1514] Processing DeleteTablet for tablet ea496383736c44b0b122139ba26de8ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:51 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:50608
I20260923 23:07:52.000584 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 18225
I20260923 23:07:52.011761 18709 tablet_replica.cc:330] T ea496383736c44b0b122139ba26de8ea P 87c9a1b65bbe46b3997f9125a4cb8590: stopping tablet replica
W20260923 23:07:52.012032 18173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:46557 (error 108)
I20260923 23:07:52.012795 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 18389
I20260923 23:07:52.016702 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:52.016550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:46557 (local address 127.0.0.1:50624)
0923 23:07:52.016604 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:52.016656 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 23:07:52.016731 18708 tablet_replica.cc:330] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: stopping tablet replica
I20260923 23:07:52.016865 18708 raft_consensus.cc:2229] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 1 LEADER]: Raft consensus shutting down.
W20260923 23:07:52.016899 18173 catalog_manager.cc:4584] TS 0fc130afa1e34611aa5e1f7a2d83945d (127.5.74.1:46557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea496383736c44b0b122139ba26de8ea: Network error: Client connection negotiation failed: client connection to 127.5.74.1:46557: connect: Connection refused (error 111)
I20260923 23:07:52.017086 18708 raft_consensus.cc:2258] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:07:52.017640 18708 ts_tablet_manager.cc:1905] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:07:52.020794 18708 ts_tablet_manager.cc:1918] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:07:52.020974 18708 log.cc:1196] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/ea496383736c44b0b122139ba26de8ea
I20260923 23:07:52.021368 18708 ts_tablet_manager.cc:1939] T ea496383736c44b0b122139ba26de8ea P 1bd2776de09e49a58a01941cd3d79ca6: Deleting consensus metadata
I20260923 23:07:52.021938 18173 catalog_manager.cc:4841] TS 1bd2776de09e49a58a01941cd3d79ca6 (127.5.74.3:44555): tablet ea496383736c44b0b122139ba26de8ea (table test-table [id=76e91ba164144ec1bf333e5cc73fa70f]) successfully deleted
I20260923 23:07:52.022642 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:52.022511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35519 (local address 127.0.0.1:39690)
0923 23:07:52.022572 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:52.022603 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.2:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 23:07:52.022765 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 18522
I20260923 23:07:52.029469 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 18154
I20260923 23:07:52.035763 5416 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 23:07:52.381186 5416 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 23:07:52.381307 5416 mini_ranger.cc:66] Stopping Ranger...
I20260923 23:07:52.726727 5416 mini_ranger.cc:68] Stopped Ranger
2026-09-23 23:07:52.726 UTC [17221] LOG: received smart shutdown request
2026-09-23 23:07:52.731 UTC [17221] LOG: background worker "logical replication launcher" (PID 17230) exited with exit code 1
2026-09-23 23:07:52.731 UTC [17225] LOG: shutting down
2026-09-23 23:07:52.957 UTC [17221] LOG: database system is shut down
2026-09-23T23:07:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50573 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:07:53 dist-test-slave-j4c5 krb5kdc[18720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:07:53 dist-test-slave-j4c5 krb5kdc[18720](info): set up 2 sockets
Sep 23 23:07:53 dist-test-slave-j4c5 krb5kdc[18720](info): commencing operation
krb5kdc: starting...
W20260923 23:07:54.140254 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:07:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:07:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:07:54.267913 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41987
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:41987,127.5.74.61:41793,127.5.74.60:44357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:54.350344 18733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:54.350622 18733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:54.350659 18733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:54.351874 18733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:07:54.351958 18733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:54.351982 18733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:54.352005 18733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:07:54.352033 18733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:07:54.353672 18733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:41987,127.5.74.61:41793,127.5.74.60:44357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:54.353996 18733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:54.354240 18733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:54.359467 18741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:54.359454 18738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:54.359454 18739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:54.359730 18733 server_base.cc:1034] running on GCE node
I20260923 23:07:54.360018 18733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:54.360409 18733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:54.361580 18733 hybrid_clock.cc:648] HybridClock initialized: now 1790204874361569 us; error 33 us; skew 500 ppm
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:54.369722 18733 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:07:54.370164 18733 webserver.cc:460] Webserver started at http://127.5.74.62:45469/ using document root <none> and password file <none>
I20260923 23:07:54.370427 18733 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:54.370476 18733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:54.370641 18733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:07:54.371791 18733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "f434cf88e88d435abe3ccd29d6b310fc"
format_stamp: "Formatted at 2026-09-23 23:07:54 on dist-test-slave-j4c5"
server_key: "b79629a1433ca91ed73a62e11725ac20"
server_key_iv: "10f41ddf061b0d66e3c7d1850d2913a1"
server_key_version: "encryptionkey@0"
I20260923 23:07:54.372196 18733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "f434cf88e88d435abe3ccd29d6b310fc"
format_stamp: "Formatted at 2026-09-23 23:07:54 on dist-test-slave-j4c5"
server_key: "b79629a1433ca91ed73a62e11725ac20"
server_key_iv: "10f41ddf061b0d66e3c7d1850d2913a1"
server_key_version: "encryptionkey@0"
I20260923 23:07:54.373832 18733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:07:54.374712 18748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:54.374941 18733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:54.375049 18733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "f434cf88e88d435abe3ccd29d6b310fc"
format_stamp: "Formatted at 2026-09-23 23:07:54 on dist-test-slave-j4c5"
server_key: "b79629a1433ca91ed73a62e11725ac20"
server_key_iv: "10f41ddf061b0d66e3c7d1850d2913a1"
server_key_version: "encryptionkey@0"
I20260923 23:07:54.375128 18733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:54.391935 18733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:54.394289 18733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:54.394562 18733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:54.398778 18733 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:41987
I20260923 23:07:54.398910 18800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:41987 every 8 connection(s)
I20260923 23:07:54.399232 18733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:07:54.400122 18801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:54.401739 18801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:07:54.403280 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 18733
I20260923 23:07:54.403441 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:07:54.403698 5416 external_mini_cluster.cc:1428] Setting key 9dbc038b69168334fd1048cb3d0f860a
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.61@KRBTEST.COM; defaulting to no policy
I20260923 23:07:54.412895 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.402652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.5.74.62:41987)
0923 23:07:54.402779 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.402785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.402823 (+ 38us) server_negotiation.cc:407] Connection header received
0923 23:07:54.403602 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.403604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.403622 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.403724 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.404359 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.405179 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.406078 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.406390 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.406519 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.408952 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.408960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.408961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.408970 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.411007 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.411477 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.411480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.411481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.411584 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.411945 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.411946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.411947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.412296 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.412368 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.412569 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.412676 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":49,"threads_started":1}
I20260923 23:07:54.413079 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.402483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41987 (local address 127.0.0.1:41234)
0923 23:07:54.402667 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.402726 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:54.403371 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:54.403728 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:54.403731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:54.404039 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:54.404262 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.404267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.405295 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.405296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.405986 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.405988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.406484 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.406485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.406548 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.407158 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:54.407171 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:54.408851 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:54.411107 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.411114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.411117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.411359 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.411707 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.411708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.411709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.411804 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.412368 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:54.412374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:54.412439 (+ 65us) client_negotiation.cc:769] Sending connection context
0923 23:07:54.412544 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 23:07:54.412950 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":193,"thread_start_us":77,"threads_started":1}
I20260923 23:07:54.413925 18801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:07:54.414287 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.414165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:41793 (local address 127.0.0.1:58134)
0923 23:07:54.414214 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.414264 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 23:07:54.414433 18750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.61@KRBTEST.COM" created.
I20260923 23:07:54.414975 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.414837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:41793 (local address 127.0.0.1:58146)
0923 23:07:54.414885 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.414922 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 23:07:54.415127 18801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.61:41793: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab.
Entry for principal kudu/127.5.74.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab.
I20260923 23:07:54.432448 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.61:41793
--webserver_interface=127.5.74.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab
--principal=kudu/127.5.74.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:41987,127.5.74.61:41793,127.5.74.60:44357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:07:54.464350 18801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } attempt: 1
I20260923 23:07:54.464954 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.464810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:41793 (local address 127.0.0.1:58158)
0923 23:07:54.464877 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.464925 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 23:07:54.465340 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.465256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:41793 (local address 127.0.0.1:58160)
0923 23:07:54.465300 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.465325 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 23:07:54.465490 18801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.61:41793: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
W20260923 23:07:54.515324 18808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:54.515558 18808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:54.515580 18808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:54.516860 18808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:07:54.516913 18808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:54.516928 18808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:54.516940 18808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:07:54.516953 18808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:07:54.518546 18808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:41987,127.5.74.61:41793,127.5.74.60:44357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab
--principal=kudu/127.5.74.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.61:41793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.74.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:54.518823 18808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:54.519061 18808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:54.524268 18816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:54.524250 18814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:54.524250 18813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:54.524279 18808 server_base.cc:1034] running on GCE node
I20260923 23:07:54.524727 18808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:54.525033 18808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:54.526203 18808 hybrid_clock.cc:648] HybridClock initialized: now 1790204874526194 us; error 41 us; skew 500 ppm
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:54.528841 18808 init.cc:377] Logged in from keytab as kudu/127.5.74.61@KRBTEST.COM (short username kudu)
I20260923 23:07:54.529340 18808 webserver.cc:460] Webserver started at http://127.5.74.61:39661/ using document root <none> and password file <none>
I20260923 23:07:54.529606 18808 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:54.529676 18808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:54.529831 18808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:07:54.530949 18808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/data/instance:
uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9"
format_stamp: "Formatted at 2026-09-23 23:07:54 on dist-test-slave-j4c5"
server_key: "5cfb7417e90cecd6ff96518456099918"
server_key_iv: "c945c6669a667c9864376eb68899126b"
server_key_version: "encryptionkey@0"
I20260923 23:07:54.531320 18808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/wal/instance:
uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9"
format_stamp: "Formatted at 2026-09-23 23:07:54 on dist-test-slave-j4c5"
server_key: "5cfb7417e90cecd6ff96518456099918"
server_key_iv: "c945c6669a667c9864376eb68899126b"
server_key_version: "encryptionkey@0"
I20260923 23:07:54.532861 18808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:07:54.533658 18801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } attempt: 2
I20260923 23:07:54.533738 18823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:54.533967 18808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:54.534101 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.533936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:41793 (local address 127.0.0.1:58162)
0923 23:07:54.534004 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.534061 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 23:07:54.534071 18808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/wal
uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9"
format_stamp: "Formatted at 2026-09-23 23:07:54 on dist-test-slave-j4c5"
server_key: "5cfb7417e90cecd6ff96518456099918"
server_key_iv: "c945c6669a667c9864376eb68899126b"
server_key_version: "encryptionkey@0"
I20260923 23:07:54.534161 18808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:54.534543 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.534415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:41793 (local address 127.0.0.1:58164)
0923 23:07:54.534488 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.534522 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 23:07:54.534632 18801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.61:41793: Network error: Client connection negotiation failed: client connection to 127.5.74.61:41793: connect: Connection refused (error 111)
I20260923 23:07:54.554008 18808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:54.556267 18808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:54.556438 18808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:54.560519 18808 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.61:41793
I20260923 23:07:54.560551 18875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.61:41793 every 8 connection(s)
I20260923 23:07:54.560952 18808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
I20260923 23:07:54.561810 18876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:54.563294 18876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:07:54.567853 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 18808
I20260923 23:07:54.568009 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-1/wal/instance
I20260923 23:07:54.568250 5416 external_mini_cluster.cc:1428] Setting key 76d15e3dc326c6fcd5bc7bae7c23b332
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:54.573153 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.564011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.5.74.62:41987)
0923 23:07:54.564104 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.564106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.564293 (+ 187us) server_negotiation.cc:407] Connection header received
0923 23:07:54.564748 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.564749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.564760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.564832 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.565721 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.566249 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.567243 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.567474 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.567534 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.569915 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.569916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.569917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.569926 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.571425 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.571975 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.571977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.571977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.572031 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.572417 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.572419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.572419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.572542 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.572611 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.573009 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.573085 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 23:07:54.573467 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.563977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41987 (local address 127.0.0.1:41236)
0923 23:07:54.564136 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.564201 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:54.564654 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:54.564917 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:54.564920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:54.565377 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:54.565616 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.565621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.566339 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.566340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.567158 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.567160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.567558 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.567559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.567665 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.568426 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:54.568434 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:54.569807 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:54.571516 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.571525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.571528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.571882 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.572103 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.572103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.572104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.572284 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.572626 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:54.572635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:54.572831 (+ 196us) client_negotiation.cc:769] Sending connection context
0923 23:07:54.572967 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 23:07:54.573169 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260923 23:07:54.574045 18876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.74.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.60@KRBTEST.COM" created.
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
I20260923 23:07:54.583031 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.574416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.5.74.61:41793)
0923 23:07:54.574595 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.574599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.574615 (+ 16us) server_negotiation.cc:407] Connection header received
0923 23:07:54.574631 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.574633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.574643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.574744 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.575029 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.575736 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.576524 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.576823 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.576916 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.579655 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.579667 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.579668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.579676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.581539 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.581953 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.581956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.581956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.582009 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.582212 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.582213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.582213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.582357 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.582421 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.582622 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.582754 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260923 23:07:54.583048 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.574335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:41793 (local address 127.0.0.1:58166)
0923 23:07:54.574389 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.574400 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:54.574443 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:54.574748 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:54.574748 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:54.574906 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:54.574950 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.574952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.575834 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.575835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.576435 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.576436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.576915 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.576916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.576983 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.577352 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:54.577356 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:54.579555 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:54.581633 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.581634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.581634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.581867 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.582077 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.582078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.582079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.582151 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.582426 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:54.582427 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:54.582480 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 23:07:54.582563 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 23:07:54.582701 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 23:07:54.583824 18876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:07:54.584259 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.584081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56472)
0923 23:07:54.584136 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.584225 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 23:07:54.584409 18825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
I20260923 23:07:54.584852 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.584736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56480)
0923 23:07:54.584792 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.584824 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 23:07:54.585007 18876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:44357: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab.
Entry for principal kudu/127.5.74.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab.
I20260923 23:07:54.596029 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.60:44357
--webserver_interface=127.5.74.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab
--principal=kudu/127.5.74.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:41987,127.5.74.61:41793,127.5.74.60:44357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:07:54.625777 18801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } attempt: 3
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
I20260923 23:07:54.633257 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.626263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.5.74.61:41793)
0923 23:07:54.626337 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.626339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.626351 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:07:54.626452 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.626454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.626466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.626534 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.626891 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.627384 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.628098 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.628411 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.628495 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.630640 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.630642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.630642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.630650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.632023 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.632444 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.632444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.632445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.632497 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.632704 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.632705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.632705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.632820 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.632902 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.633077 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.633158 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:07:54.633257 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.626187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:41793 (local address 127.0.0.1:58168)
0923 23:07:54.626264 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.626280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:54.626352 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:54.626562 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:54.626563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:54.626728 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:54.626793 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.626795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.627486 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.627487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.628007 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.628009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.628493 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.628494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.628582 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.629098 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:54.629102 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:54.630545 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:54.632121 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.632121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.632122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.632333 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.632570 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.632571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.632571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.632631 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.632890 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:54.632891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:54.632961 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 23:07:54.633051 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 23:07:54.633157 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 23:07:54.633980 18801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:07:54.634217 18876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } attempt: 1
I20260923 23:07:54.634380 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.634246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56484)
0923 23:07:54.634308 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.634349 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 23:07:54.634528 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.634431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56488)
0923 23:07:54.634472 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.634502 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 23:07:54.634685 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.634597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56498)
0923 23:07:54.634636 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.634658 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:07:54.634833 18801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:44357: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
I20260923 23:07:54.634985 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.634893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56514)
0923 23:07:54.634936 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.634957 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 23:07:54.635062 18876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:44357: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
I20260923 23:07:54.666030 18801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } attempt: 1
I20260923 23:07:54.666575 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.666424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56530)
0923 23:07:54.666493 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.666539 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 23:07:54.666991 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.666888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56542)
0923 23:07:54.666938 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.666969 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:07:54.667115 18801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:44357: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
W20260923 23:07:54.681196 18883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:54.681412 18883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:54.681435 18883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:54.682595 18883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:07:54.682686 18883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:54.682703 18883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:54.682715 18883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:07:54.682727 18883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:07:54.684374 18883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:41987,127.5.74.61:41793,127.5.74.60:44357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab
--principal=kudu/127.5.74.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.60:44357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.74.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:54.684628 18883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:54.684867 18883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:54.690025 18891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:54.690003 18889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:54.689991 18888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:54.690155 18883 server_base.cc:1034] running on GCE node
I20260923 23:07:54.690637 18883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:54.690958 18883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:54.692255 18883 hybrid_clock.cc:648] HybridClock initialized: now 1790204874692236 us; error 27 us; skew 500 ppm
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:54.695055 18883 init.cc:377] Logged in from keytab as kudu/127.5.74.60@KRBTEST.COM (short username kudu)
I20260923 23:07:54.695616 18883 webserver.cc:460] Webserver started at http://127.5.74.60:44029/ using document root <none> and password file <none>
I20260923 23:07:54.695895 18883 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:54.695986 18883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:54.696195 18883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:07:54.697412 18883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/data/instance:
uuid: "650538b2f2f5408aac47951dd7c95de9"
format_stamp: "Formatted at 2026-09-23 23:07:54 on dist-test-slave-j4c5"
server_key: "d20560a0caf04913d3a48313533de392"
server_key_iv: "f70fc5f665cad65b386586642ac43542"
server_key_version: "encryptionkey@0"
I20260923 23:07:54.697803 18883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/wal/instance:
uuid: "650538b2f2f5408aac47951dd7c95de9"
format_stamp: "Formatted at 2026-09-23 23:07:54 on dist-test-slave-j4c5"
server_key: "d20560a0caf04913d3a48313533de392"
server_key_iv: "f70fc5f665cad65b386586642ac43542"
server_key_version: "encryptionkey@0"
I20260923 23:07:54.699435 18883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:07:54.700366 18898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:54.700554 18883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:54.700665 18883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/wal
uuid: "650538b2f2f5408aac47951dd7c95de9"
format_stamp: "Formatted at 2026-09-23 23:07:54 on dist-test-slave-j4c5"
server_key: "d20560a0caf04913d3a48313533de392"
server_key_iv: "f70fc5f665cad65b386586642ac43542"
server_key_version: "encryptionkey@0"
I20260923 23:07:54.700752 18883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:54.703218 18876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } attempt: 2
I20260923 23:07:54.703642 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.703513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56546)
0923 23:07:54.703574 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.703614 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 23:07:54.703975 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.703876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56562)
0923 23:07:54.703922 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.703947 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 23:07:54.704085 18876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:44357: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
I20260923 23:07:54.741882 18883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:54.742235 18801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } attempt: 2
I20260923 23:07:54.742668 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.742532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56570)
0923 23:07:54.742599 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.742638 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 23:07:54.743006 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.742915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56580)
0923 23:07:54.742959 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.742982 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 23:07:54.743297 18801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:44357: Network error: Client connection negotiation failed: client connection to 127.5.74.60:44357: connect: Connection refused (error 111)
I20260923 23:07:54.744405 18883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:54.744570 18883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:54.748441 18883 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.60:44357
I20260923 23:07:54.748513 18950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.60:44357 every 8 connection(s)
I20260923 23:07:54.748893 18883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
I20260923 23:07:54.749833 18951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:54.751410 18951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:07:54.751767 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 18883
I20260923 23:07:54.751911 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204714633713-5416-0/minicluster-data/master-2/wal/instance
I20260923 23:07:54.752209 5416 external_mini_cluster.cc:1428] Setting key f82f4a8ae0da6339f98ea9397917c9b8
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
I20260923 23:07:54.761155 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.752264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.5.74.62:41987)
0923 23:07:54.752321 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.752324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.752626 (+ 302us) server_negotiation.cc:407] Connection header received
0923 23:07:54.753231 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.753233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.753245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.753307 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.753879 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.754359 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.755273 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.755487 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.755576 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.757995 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.757996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.757997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.758005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.759716 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.760106 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.760108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.760108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.760203 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.760482 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.760484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.760484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.760617 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.760698 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.760981 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.761061 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:07:54.761350 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.752262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:41987 (local address 127.0.0.1:41250)
0923 23:07:54.752457 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.752496 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:54.753125 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:54.753328 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:54.753331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:54.753611 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:54.753763 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.753767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.754493 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.754494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.755181 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.755183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.755585 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.755587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.755749 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.756344 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:54.756355 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:54.757908 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:54.759800 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.759805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.759808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.760027 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.760278 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.760279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.760279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.760379 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.760689 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:54.760695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:54.760835 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 23:07:54.760946 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 23:07:54.761100 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260923 23:07:54.761529 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.752577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.5.74.60:44357)
0923 23:07:54.752720 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.752725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.752820 (+ 95us) server_negotiation.cc:407] Connection header received
0923 23:07:54.753139 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.753150 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.753172 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.753280 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.753930 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.754714 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.755474 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.755785 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.755879 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.758657 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.758665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.758665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.758673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.760356 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.760703 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.760706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.760707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.760764 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.760973 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.760974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.760975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.761119 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.761283 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.761328 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.761409 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260923 23:07:54.762024 18951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:54.769811 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.762302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:41793 (local address 127.0.0.1:58182)
0923 23:07:54.762339 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.762366 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:54.762436 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:54.762608 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:54.762609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:54.762748 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:54.762811 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.762813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.763661 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.763662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.764364 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.764366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.764856 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.764856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.764915 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.765330 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:54.765335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:54.766808 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:54.768626 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.768627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.768628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.768878 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.769085 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.769086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.769086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.769152 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.769453 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:54.769454 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:54.769507 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 23:07:54.769594 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 23:07:54.769696 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 23:07:54.769817 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.762428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.5.74.61:41793)
0923 23:07:54.762496 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.762498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.762507 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:07:54.762533 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.762534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.762544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.762605 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.762885 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.763588 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.764439 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.764760 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.764842 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.766901 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.766903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.766904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.766912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.768543 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.768961 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.768961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.768962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.769014 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.769225 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.769226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.769226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.769371 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.769452 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.769620 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.769719 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 23:07:54.770221 18951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:07:54.771150 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.762689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.5.74.62:41987)
0923 23:07:54.762750 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.762751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.762759 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:07:54.762777 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.762778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.762789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.762857 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.763441 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.764079 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.765137 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.765363 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.765455 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.767836 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.767838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.767838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.767846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.769592 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.770034 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.770035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.770035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.770095 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.770307 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.770307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.770308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.770495 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.770662 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.770889 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.770984 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
I20260923 23:07:54.777765 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.770511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56606)
0923 23:07:54.770607 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.770618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:54.770670 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:54.770813 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:54.770814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:54.770953 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:54.770982 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.770984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.771823 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.771824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.772384 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.772386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.772787 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.772788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.772836 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.773279 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:54.773283 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:54.774748 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:54.776523 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.776524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.776524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.776766 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.776997 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.776998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.776998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.777054 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.777343 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:54.777344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:54.777394 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 23:07:54.777486 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 23:07:54.777595 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 23:07:54.777828 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.770569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.5.74.60:44357)
0923 23:07:54.770609 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.770611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.770654 (+ 43us) server_negotiation.cc:407] Connection header received
0923 23:07:54.770737 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.770738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.770749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.770811 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.771064 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.771732 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.772482 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.772695 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.772787 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.774848 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.774850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.774850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.774858 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.776438 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.776842 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.776843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.776844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.776921 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.777120 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.777121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.777122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.777262 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.777313 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.777509 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.777659 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":19}
I20260923 23:07:54.779597 18951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9: Bootstrap starting.
I20260923 23:07:54.780392 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.771842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.5.74.61:41793)
0923 23:07:54.771880 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.771881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.771924 (+ 43us) server_negotiation.cc:407] Connection header received
0923 23:07:54.771997 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.771998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.772007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.772061 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.772586 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.773032 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.773991 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.774166 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.774233 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.777210 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.777211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.777211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.777218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.779179 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.779514 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.779515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.779515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.779574 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.779757 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.779758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.779758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.779886 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.780091 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.780175 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.780280 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:07:54.780728 18951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:54.781061 18951 log.cc:824] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:54.781940 18951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9: No bootstrap required, opened a new log
I20260923 23:07:54.782226 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:07:54.783887 18951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } }
I20260923 23:07:54.784108 18951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:54.784224 18951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650538b2f2f5408aac47951dd7c95de9, State: Initialized, Role: FOLLOWER
I20260923 23:07:54.784399 18951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } }
I20260923 23:07:54.784749 18951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:07:54.786866 18964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:54.786978 18964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:07:54.786576 18975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:07:54.787101 18975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:07:54.787459 18883 master_runner.cc:186] Error getting master registration for 127.5.74.62:41987: OK, instance_id { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" instance_seqno: 1790204874397549 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:07:54.790107 18883 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:07:54.791464 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.783635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.5.74.62:41987)
0923 23:07:54.783687 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.783689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.783746 (+ 57us) server_negotiation.cc:407] Connection header received
0923 23:07:54.783821 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.783822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.783832 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.783897 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.784715 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.785461 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.786522 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.786848 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.786895 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.788552 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.788553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.788553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.788560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.790183 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.790557 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.790558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.790559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.790641 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.790843 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.790844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.790845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.790994 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.791112 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.791270 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.791363 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:07:54.793843 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.783942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.5.74.60:44357)
0923 23:07:54.784012 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.784014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.785077 (+ 1063us) server_negotiation.cc:407] Connection header received
0923 23:07:54.785165 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.785166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.785175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.785205 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.785890 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.786588 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.788631 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.789270 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.789316 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.790966 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.790967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.790967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.790974 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.792756 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.793142 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.793142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.793143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.793191 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.793389 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.793389 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.793390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.793508 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.793581 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.793719 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.793780 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 23:07:54.795230 18876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } attempt: 3
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
I20260923 23:07:54.800585 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.784006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.5.74.61:41793)
0923 23:07:54.784046 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.784048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.784126 (+ 78us) server_negotiation.cc:407] Connection header received
0923 23:07:54.784241 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.784242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.784251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.784278 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.784982 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.785743 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.786854 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.787121 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.787162 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.797021 (+ 9859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.797022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.797022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.797029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.798917 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.799315 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.799317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.799317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.799372 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.799591 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.799591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.799592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.799706 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.800391 (+ 685us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.800436 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.800499 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:07:54.803432 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.795532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56626)
0923 23:07:54.795662 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.795674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:54.795731 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:54.795860 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:54.795861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:54.796045 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:54.796091 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.796094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.796800 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.796801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.797597 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.797601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.798086 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.798087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.798152 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.798500 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:54.798505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:54.800258 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:54.802015 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.802016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.802017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.802216 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.802663 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.802664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.802664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.802747 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.803099 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:54.803100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:54.803149 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 23:07:54.803229 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 23:07:54.803328 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260923 23:07:54.803522 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.795610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.5.74.60:44357)
0923 23:07:54.795664 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.795665 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.795711 (+ 46us) server_negotiation.cc:407] Connection header received
0923 23:07:54.795796 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.795797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.795805 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.795846 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.796204 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.796710 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.797706 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.797994 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.798073 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.800355 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.800357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.800357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.800364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.801929 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.802294 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.802295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.802295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.802356 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.802893 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.802894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.802895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.803017 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.803091 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.803249 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.803416 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:07:54.805490 18876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9: Bootstrap starting.
I20260923 23:07:54.806335 18876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:54.806663 18876 log.cc:824] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:54.807505 18876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9: No bootstrap required, opened a new log
I20260923 23:07:54.809162 18876 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } }
I20260923 23:07:54.809286 18876 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:54.809316 18876 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2da9bb50ef443e0ac1f5a9302d0feb9, State: Initialized, Role: FOLLOWER
I20260923 23:07:54.809484 18876 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } }
I20260923 23:07:54.809818 18876 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:07:54.809787 18978 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:54.809902 18978 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:07:54.811787 18989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:07:54.811848 18989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:07:54.811860 18808 master_runner.cc:186] Error getting master registration for 127.5.74.62:41987: OK, instance_id { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" instance_seqno: 1790204874397549 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:07:54.814208 18808 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:07:54.842469 18801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } attempt: 3
Sep 23 23:07:54 dist-test-slave-j4c5 krb5kdc[18720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
I20260923 23:07:54.849898 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.842926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.5.74.60:44357)
0923 23:07:54.842992 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:54.842994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:54.843005 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:07:54.843062 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:54.843063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:54.843075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:54.843155 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:54.843494 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.843982 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.844792 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.844981 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.845074 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.847091 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:54.847093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:54.847093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:54.847109 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:54.848647 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.849056 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.849058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.849058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.849114 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:54.849332 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:54.849333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:54.849334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:54.849456 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:54.849543 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:54.849701 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 23:07:54.849785 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:07:54.849892 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:54.842870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:44357 (local address 127.0.0.1:56642)
0923 23:07:54.842935 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:54.842948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:54.842997 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:54.843160 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:54.843161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:54.843353 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:54.843400 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.843402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.844081 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.844082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.844682 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:54.844685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:54.845085 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:54.845085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:54.845154 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:54.845699 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:54.845704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:54.846983 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:54.848760 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.848761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.848761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.848949 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.849192 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:54.849192 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:54.849193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:54.849264 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:54.849534 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:54.849535 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:54.849584 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 23:07:54.849676 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 23:07:54.849785 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 23:07:54.852480 18801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc: Bootstrap starting.
I20260923 23:07:54.853147 18801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:54.853449 18801 log.cc:824] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:54.854245 18801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc: No bootstrap required, opened a new log
I20260923 23:07:54.855705 18801 raft_consensus.cc:348] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } }
I20260923 23:07:54.855861 18801 raft_consensus.cc:374] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:54.855947 18801 raft_consensus.cc:729] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f434cf88e88d435abe3ccd29d6b310fc, State: Initialized, Role: FOLLOWER
I20260923 23:07:54.856123 18801 consensus_queue.cc:260] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } }
I20260923 23:07:54.856473 18992 sys_catalog.cc:455] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:54.856537 18801 sys_catalog.cc:564] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:07:54.856622 18992 sys_catalog.cc:458] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:07:54.858584 19003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:07:54.858698 19003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:07:54.861095 18733 master_runner.cc:263] No leader master found from master f434cf88e88d435abe3ccd29d6b310fc
I20260923 23:07:54.861145 18733 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 23:07:54.895465 18754 tablet.cc:2369] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:07:54.995241 18904 tablet.cc:2369] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:07:55.027609 18978 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:07:55.027753 18978 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } }
I20260923 23:07:55.028060 18978 leader_election.cc:288] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f434cf88e88d435abe3ccd29d6b310fc (127.5.74.62:41987), 650538b2f2f5408aac47951dd7c95de9 (127.5.74.60:44357)
I20260923 23:07:55.028512 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f434cf88e88d435abe3ccd29d6b310fc" is_pre_election: true
I20260923 23:07:55.028644 18775 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2da9bb50ef443e0ac1f5a9302d0feb9 in term 0.
I20260923 23:07:55.028656 18925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "650538b2f2f5408aac47951dd7c95de9" is_pre_election: true
I20260923 23:07:55.028854 18925 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2da9bb50ef443e0ac1f5a9302d0feb9 in term 0.
I20260923 23:07:55.028872 18825 leader_election.cc:302] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2da9bb50ef443e0ac1f5a9302d0feb9, f434cf88e88d435abe3ccd29d6b310fc; no voters:
I20260923 23:07:55.029054 18978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:07:55.029112 18978 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:07:55.029129 18978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:55.029879 18978 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } }
I20260923 23:07:55.029987 18978 leader_election.cc:288] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [CANDIDATE]: Term 1 election: Requested vote from peers f434cf88e88d435abe3ccd29d6b310fc (127.5.74.62:41987), 650538b2f2f5408aac47951dd7c95de9 (127.5.74.60:44357)
I20260923 23:07:55.030248 18925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "650538b2f2f5408aac47951dd7c95de9"
I20260923 23:07:55.030273 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f434cf88e88d435abe3ccd29d6b310fc"
I20260923 23:07:55.030324 18925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:55.030417 18775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:55.031083 18925 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2da9bb50ef443e0ac1f5a9302d0feb9 in term 1.
I20260923 23:07:55.031255 18825 leader_election.cc:302] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 650538b2f2f5408aac47951dd7c95de9, c2da9bb50ef443e0ac1f5a9302d0feb9; no voters:
I20260923 23:07:55.031297 18775 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2da9bb50ef443e0ac1f5a9302d0feb9 in term 1.
I20260923 23:07:55.031363 18978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:07:55.031533 18978 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [term 1 LEADER]: Becoming Leader. State: Replica: c2da9bb50ef443e0ac1f5a9302d0feb9, State: Running, Role: LEADER
I20260923 23:07:55.031641 18978 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } }
I20260923 23:07:55.032195 19005 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2da9bb50ef443e0ac1f5a9302d0feb9. Latest consensus state: current_term: 1 leader_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:55.032318 19005 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:55.032518 19007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:07:55.032708 19007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:07:55.034137 18775 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [term 1 FOLLOWER]: Refusing update from remote peer c2da9bb50ef443e0ac1f5a9302d0feb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:07:55.034456 18925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [term 1 FOLLOWER]: Refusing update from remote peer c2da9bb50ef443e0ac1f5a9302d0feb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:07:55.034488 18978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:07:55.034747 18978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:07:55.035732 18992 sys_catalog.cc:455] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2da9bb50ef443e0ac1f5a9302d0feb9. Latest consensus state: current_term: 1 leader_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:55.035849 18992 sys_catalog.cc:458] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:07:55.035876 18964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2da9bb50ef443e0ac1f5a9302d0feb9. Latest consensus state: current_term: 1 leader_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:55.035966 18964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:07:55.037042 18978 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:55.037055 19013 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:55.037199 19013 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:55.037201 18992 sys_catalog.cc:455] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:55.037264 18992 sys_catalog.cc:458] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:07:55.037144 18978 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:55.037370 18964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f434cf88e88d435abe3ccd29d6b310fc" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41987 } } peers { permanent_uuid: "c2da9bb50ef443e0ac1f5a9302d0feb9" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 41793 } } peers { permanent_uuid: "650538b2f2f5408aac47951dd7c95de9" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 44357 } } }
I20260923 23:07:55.037460 18964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:07:55.037602 19009 mvcc.cc:205] Tried to move back new op lower bound from 7332679168137318400 to 7332679168130334720. Current Snapshot: MvccSnapshot[applied={T|T < 7332679168137318400}]
I20260923 23:07:55.037657 19007 catalog_manager.cc:1347] Generated new cluster ID: 8144f32da83d461f8d9a308e87f96b37
I20260923 23:07:55.037694 19007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:07:55.037698 19010 mvcc.cc:205] Tried to move back new op lower bound from 7332679168137318400 to 7332679168130334720. Current Snapshot: MvccSnapshot[applied={T|T < 7332679168137318400}]
I20260923 23:07:55.048137 19007 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:07:55.048882 19007 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:07:55.055195 19007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2da9bb50ef443e0ac1f5a9302d0feb9: Generated new TSK 0
I20260923 23:07:55.055486 19007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:07:55.787595 18975 catalog_manager.cc:1259] Loaded cluster ID: 8144f32da83d461f8d9a308e87f96b37
I20260923 23:07:55.787700 18975 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9: loading cluster ID for follower catalog manager: success
I20260923 23:07:55.788555 18975 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9: acquiring CA information for follower catalog manager: success
I20260923 23:07:55.788894 18975 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 650538b2f2f5408aac47951dd7c95de9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:07:55.859187 19003 catalog_manager.cc:1259] Loaded cluster ID: 8144f32da83d461f8d9a308e87f96b37
I20260923 23:07:55.859269 19003 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc: loading cluster ID for follower catalog manager: success
I20260923 23:07:55.860140 19003 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc: acquiring CA information for follower catalog manager: success
I20260923 23:07:55.860533 19003 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f434cf88e88d435abe3ccd29d6b310fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:07:56.117699 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 18733
I20260923 23:07:56.123409 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 18808
I20260923 23:07:56.129359 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 18883
2026-09-23T23:07:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3062 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:07:56 dist-test-slave-j4c5 krb5kdc[19023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:07:56 dist-test-slave-j4c5 krb5kdc[19023](info): set up 2 sockets
Sep 23 23:07:56 dist-test-slave-j4c5 krb5kdc[19023](info): commencing operation
krb5kdc: starting...
W20260923 23:07:57.208345 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:07:57 dist-test-slave-j4c5 krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:07:57.340637 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41509
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:41509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:57.432715 19036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:57.432928 19036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:57.432952 19036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:57.434079 19036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:07:57.434120 19036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:57.434134 19036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:57.434146 19036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:07:57.434158 19036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:07:57.435586 19036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:41509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:41509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:57.435791 19036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:57.436072 19036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:57.441231 19041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:57.441310 19044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:57.441358 19042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:57.441624 19036 server_base.cc:1034] running on GCE node
I20260923 23:07:57.441891 19036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:57.442219 19036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:57.443398 19036 hybrid_clock.cc:648] HybridClock initialized: now 1790204877443390 us; error 52 us; skew 500 ppm
Sep 23 23:07:57 dist-test-slave-j4c5 krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204877, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:57.446308 19036 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:07:57.446990 19036 webserver.cc:460] Webserver started at http://127.5.74.62:42189/ using document root <none> and password file <none>
I20260923 23:07:57.447243 19036 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:57.447315 19036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:57.447451 19036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:07:57.448622 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "b569ce6e461248ee8fb491b755024c45"
format_stamp: "Formatted at 2026-09-23 23:07:57 on dist-test-slave-j4c5"
server_key: "20d7ffd074d57ffbf45ddcf21dc6ac22"
server_key_iv: "7121972bf7a08a4147e6f5e5b088fbaa"
server_key_version: "encryptionkey@0"
I20260923 23:07:57.448999 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "b569ce6e461248ee8fb491b755024c45"
format_stamp: "Formatted at 2026-09-23 23:07:57 on dist-test-slave-j4c5"
server_key: "20d7ffd074d57ffbf45ddcf21dc6ac22"
server_key_iv: "7121972bf7a08a4147e6f5e5b088fbaa"
server_key_version: "encryptionkey@0"
I20260923 23:07:57.450531 19036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:07:57.451462 19051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:57.451822 19036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:57.451943 19036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "b569ce6e461248ee8fb491b755024c45"
format_stamp: "Formatted at 2026-09-23 23:07:57 on dist-test-slave-j4c5"
server_key: "20d7ffd074d57ffbf45ddcf21dc6ac22"
server_key_iv: "7121972bf7a08a4147e6f5e5b088fbaa"
server_key_version: "encryptionkey@0"
I20260923 23:07:57.452039 19036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:57.469569 19036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:57.471876 19036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:57.472067 19036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:57.476464 19036 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:41509
I20260923 23:07:57.476492 19103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:41509 every 8 connection(s)
I20260923 23:07:57.476879 19036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:07:57.477741 19104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:57.480002 19104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45: Bootstrap starting.
I20260923 23:07:57.480636 19104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:57.480880 19104 log.cc:824] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:57.481594 19104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45: No bootstrap required, opened a new log
I20260923 23:07:57.483507 19104 raft_consensus.cc:348] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569ce6e461248ee8fb491b755024c45" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41509 } }
I20260923 23:07:57.483656 19104 raft_consensus.cc:374] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:57.483680 19104 raft_consensus.cc:729] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b569ce6e461248ee8fb491b755024c45, State: Initialized, Role: FOLLOWER
I20260923 23:07:57.483836 19104 consensus_queue.cc:260] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569ce6e461248ee8fb491b755024c45" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41509 } }
I20260923 23:07:57.483951 19104 raft_consensus.cc:388] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:07:57.483976 19104 raft_consensus.cc:482] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:07:57.484005 19104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:57.484629 19104 raft_consensus.cc:504] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569ce6e461248ee8fb491b755024c45" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41509 } }
I20260923 23:07:57.484719 19104 leader_election.cc:302] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b569ce6e461248ee8fb491b755024c45; no voters:
I20260923 23:07:57.484858 19104 leader_election.cc:288] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:07:57.484943 19107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:07:57.485134 19107 raft_consensus.cc:686] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [term 1 LEADER]: Becoming Leader. State: Replica: b569ce6e461248ee8fb491b755024c45, State: Running, Role: LEADER
I20260923 23:07:57.485157 19104 sys_catalog.cc:564] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:07:57.485224 19107 consensus_queue.cc:237] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569ce6e461248ee8fb491b755024c45" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41509 } }
I20260923 23:07:57.485797 19108 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b569ce6e461248ee8fb491b755024c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569ce6e461248ee8fb491b755024c45" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41509 } } }
I20260923 23:07:57.486016 19108 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:57.485949 19109 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b569ce6e461248ee8fb491b755024c45. Latest consensus state: current_term: 1 leader_uuid: "b569ce6e461248ee8fb491b755024c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569ce6e461248ee8fb491b755024c45" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 41509 } } }
I20260923 23:07:57.486202 19109 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:57.486461 19117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:07:57.486835 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 19036
I20260923 23:07:57.486940 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:07:57.486990 19117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:07:57.487169 5416 external_mini_cluster.cc:1428] Setting key 0afdd5fa5eff55d1de77f6d837ec8608
I20260923 23:07:57.488565 19117 catalog_manager.cc:1347] Generated new cluster ID: 9126150ebf3b46d58dd5c2a2b310b0de
I20260923 23:07:57.488619 19117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:07:57 dist-test-slave-j4c5 krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:57.497731 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:57.487625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.5.74.62:41509)
0923 23:07:57.487773 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:57.487783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:57.488261 (+ 478us) server_negotiation.cc:407] Connection header received
0923 23:07:57.488401 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:57.488408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:57.488467 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:57.488606 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:57.489341 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:57.490273 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:57.491258 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:57.491705 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:57.491832 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:57.494231 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:57.494236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:57.494236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:57.494249 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:57.496013 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:57.496454 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:57.496458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:57.496458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:57.496570 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:57.496795 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:57.496796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:57.496796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:57.497065 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:57.497252 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:57.497333 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 23:07:57.497501 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260923 23:07:57.498822 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:07:57.510564 19117 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:07:57.511163 19117 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:07:57.513561 19117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b569ce6e461248ee8fb491b755024c45: Generated new TSK 0
I20260923 23:07:57.513820 19117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:07:57.553227 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 19036
2026-09-23T23:07:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1422 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:07:57 dist-test-slave-j4c5 krb5kdc[19131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:07:57 dist-test-slave-j4c5 krb5kdc[19131](info): set up 2 sockets
Sep 23 23:07:57 dist-test-slave-j4c5 krb5kdc[19131](info): commencing operation
krb5kdc: starting...
W20260923 23:07:58.620808 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:07:58 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:07:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:07:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:07:58.748826 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34511
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:58.834339 19144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:58.834636 19144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:58.834681 19144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:58.836009 19144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:07:58.836098 19144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:58.836122 19144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:58.836143 19144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:07:58.836162 19144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:07:58.837869 19144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:34511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:58.838198 19144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:58.838654 19144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:58.844035 19150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:58.843976 19152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:58.843976 19149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:58.844290 19144 server_base.cc:1034] running on GCE node
I20260923 23:07:58.844586 19144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:58.844856 19144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:58.846024 19144 hybrid_clock.cc:648] HybridClock initialized: now 1790204878846010 us; error 33 us; skew 500 ppm
Sep 23 23:07:58 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204878, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:58.849130 19144 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:07:58.849797 19144 webserver.cc:460] Webserver started at http://127.5.74.62:33721/ using document root <none> and password file <none>
I20260923 23:07:58.850078 19144 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:58.850128 19144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:58.850282 19144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:07:58.851446 19144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "0a49c2770989485bb6be45d723bfa657"
format_stamp: "Formatted at 2026-09-23 23:07:58 on dist-test-slave-j4c5"
server_key: "3c30f4757030aa1a13ecddd0095a8cae"
server_key_iv: "1310892f7c2be5b525f3515688a67ffd"
server_key_version: "encryptionkey@0"
I20260923 23:07:58.851819 19144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "0a49c2770989485bb6be45d723bfa657"
format_stamp: "Formatted at 2026-09-23 23:07:58 on dist-test-slave-j4c5"
server_key: "3c30f4757030aa1a13ecddd0095a8cae"
server_key_iv: "1310892f7c2be5b525f3515688a67ffd"
server_key_version: "encryptionkey@0"
I20260923 23:07:58.853456 19144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:07:58.854507 19159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:58.854756 19144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:07:58.854897 19144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "0a49c2770989485bb6be45d723bfa657"
format_stamp: "Formatted at 2026-09-23 23:07:58 on dist-test-slave-j4c5"
server_key: "3c30f4757030aa1a13ecddd0095a8cae"
server_key_iv: "1310892f7c2be5b525f3515688a67ffd"
server_key_version: "encryptionkey@0"
I20260923 23:07:58.854987 19144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:58.868897 19144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:58.871029 19144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:58.871212 19144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:58.875773 19144 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:34511
I20260923 23:07:58.875907 19211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:34511 every 8 connection(s)
I20260923 23:07:58.876309 19144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:07:58.877415 19212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:58.879982 19212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Bootstrap starting.
I20260923 23:07:58.880664 19212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:58.880915 19212 log.cc:824] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:58.881832 19212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: No bootstrap required, opened a new log
I20260923 23:07:58.883461 19212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:07:58.883612 19212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:58.883635 19212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a49c2770989485bb6be45d723bfa657, State: Initialized, Role: FOLLOWER
I20260923 23:07:58.883795 19212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:07:58.883885 19212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:07:58.883910 19212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:07:58.883940 19212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:58.884763 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 19144
I20260923 23:07:58.884897 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:07:58.885061 19212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:07:58.885134 5416 external_mini_cluster.cc:1428] Setting key 161ade5f5a1a803039c6f7fa2370a684
I20260923 23:07:58.885182 19212 leader_election.cc:302] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a49c2770989485bb6be45d723bfa657; no voters:
I20260923 23:07:58.885396 19212 leader_election.cc:288] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:07:58.885474 19215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:07:58.885593 19215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 1 LEADER]: Becoming Leader. State: Replica: 0a49c2770989485bb6be45d723bfa657, State: Running, Role: LEADER
I20260923 23:07:58.885705 19212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:07:58.885732 19215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:07:58.886119 19215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a49c2770989485bb6be45d723bfa657. Latest consensus state: current_term: 1 leader_uuid: "0a49c2770989485bb6be45d723bfa657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } } }
I20260923 23:07:58.886189 19215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:58.886351 19216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a49c2770989485bb6be45d723bfa657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } } }
I20260923 23:07:58.886523 19216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:58.886983 19221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:07:58.887537 19221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:07:58.889190 19221 catalog_manager.cc:1347] Generated new cluster ID: e605a38d855b4a4b9177a3d180057a2a
I20260923 23:07:58.889245 19221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:07:58 dist-test-slave-j4c5 krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:58.897729 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:58.885868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.5.74.62:34511)
0923 23:07:58.886357 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:58.886367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:58.886408 (+ 41us) server_negotiation.cc:407] Connection header received
0923 23:07:58.886551 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:58.886560 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:58.886627 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:58.886803 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:58.887584 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:58.888932 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:58.890032 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:58.890645 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:58.890756 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:58.893635 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:58.893640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:58.893641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:58.893654 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:58.895860 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:58.896308 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:58.896312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:58.896312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:58.896457 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:58.896683 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:58.896684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:58.896685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:58.897048 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:58.897213 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:58.897296 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 23:07:58.897496 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":100,"threads_started":1}
I20260923 23:07:58.925933 19221 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:07:58.926548 19221 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:07:58.934881 19221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Generated new TSK 0
I20260923 23:07:58.935117 19221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:07:58.977046 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34511
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:59.065795 19238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:59.066038 19238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:59.066064 19238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:59.067341 19238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:59.067395 19238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:59.067425 19238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:07:59.069080 19238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:59.069381 19238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:59.069636 19238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:59.075420 19244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:59.075572 19243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:59.075495 19246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:59.075700 19238 server_base.cc:1034] running on GCE node
I20260923 23:07:59.075987 19238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:59.076299 19238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:59.077495 19238 hybrid_clock.cc:648] HybridClock initialized: now 1790204879077473 us; error 37 us; skew 500 ppm
Sep 23 23:07:59 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:59.080453 19238 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:07:59.081013 19238 webserver.cc:460] Webserver started at http://127.5.74.1:39545/ using document root <none> and password file <none>
I20260923 23:07:59.081305 19238 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:59.081388 19238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:59.081549 19238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:07:59.082741 19238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "b0f583019a264222a3479f05968603c1"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "b854b8ae16eb4b39ef2d6df0c9be799f"
server_key_iv: "e40db2cbc8a046f7f19f5afa7930b159"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.083164 19238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "b0f583019a264222a3479f05968603c1"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "b854b8ae16eb4b39ef2d6df0c9be799f"
server_key_iv: "e40db2cbc8a046f7f19f5afa7930b159"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.084798 19238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 23:07:59.086022 19253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.086283 19238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:59.086369 19238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "b0f583019a264222a3479f05968603c1"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "b854b8ae16eb4b39ef2d6df0c9be799f"
server_key_iv: "e40db2cbc8a046f7f19f5afa7930b159"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.086510 19238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:59.101512 19238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:59.104127 19238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:59.104367 19238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:59.104595 19238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:59.104943 19238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:59.105017 19238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.105042 19238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:59.105055 19238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.111348 19238 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:33655
I20260923 23:07:59.111392 19366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:33655 every 8 connection(s)
I20260923 23:07:59.111819 19238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:07:59.112725 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 19238
I20260923 23:07:59.112875 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:07:59.113131 5416 external_mini_cluster.cc:1428] Setting key 927e92843cc16113c50747dae39453b5
Sep 23 23:07:59 dist-test-slave-j4c5 krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:07:59.124641 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.112808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:53341 (local address 127.5.74.62:34511)
0923 23:07:59.112876 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.112878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.113127 (+ 249us) server_negotiation.cc:407] Connection header received
0923 23:07:59.113752 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.113753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.113769 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.113821 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:59.114991 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.115926 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.117136 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.117482 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.117568 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.120425 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:59.120427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:59.120427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:59.120437 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:59.122565 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.123223 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.123224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.123225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.123311 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.123633 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.123634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.123634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.123832 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:59.123922 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.124345 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.124508 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 23:07:59.125186 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.112785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:53341)
0923 23:07:59.112978 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.113036 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.113627 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.113906 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.113910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.114555 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:59.114844 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.114850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.116069 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.116070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.117036 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.117038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.117577 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.117578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.117705 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.118348 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:59.118358 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:59.120297 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:59.122672 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:59.122683 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:59.122686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:59.123120 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:59.123368 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:59.123369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:59.123369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:59.123563 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:59.123927 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:59.123933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:59.124132 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.124323 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.124530 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260923 23:07:59.125921 19367 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34511
I20260923 23:07:59.126040 19367 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:59.126294 19367 heartbeater.cc:511] Master 127.5.74.62:34511 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
I20260923 23:07:59.126995 19176 ts_manager.cc:194] Registered new tserver with Master: b0f583019a264222a3479f05968603c1 (127.5.74.1:33655)
I20260923 23:07:59.127955 19176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:53341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:07:59.144811 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34511
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:59.230283 19372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:59.230515 19372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:59.230541 19372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:59.231815 19372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:59.231873 19372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:59.231904 19372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:07:59.233543 19372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:59.233845 19372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:59.234107 19372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:59.239733 19377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:59.239737 19378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:59.239733 19380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:59.240031 19372 server_base.cc:1034] running on GCE node
I20260923 23:07:59.240299 19372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:59.240602 19372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:59.241884 19372 hybrid_clock.cc:648] HybridClock initialized: now 1790204879241865 us; error 27 us; skew 500 ppm
Sep 23 23:07:59 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:59.244735 19372 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:07:59.245307 19372 webserver.cc:460] Webserver started at http://127.5.74.2:46275/ using document root <none> and password file <none>
I20260923 23:07:59.245571 19372 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:59.245643 19372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:59.245805 19372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:07:59.247020 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "f0587c36654545f3850069ba68f58d61"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "16da212634d58093162c891d6e85dc7b"
server_key_iv: "56c330984f6f8df01f64a827b0fce42d"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.247653 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "f0587c36654545f3850069ba68f58d61"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "16da212634d58093162c891d6e85dc7b"
server_key_iv: "56c330984f6f8df01f64a827b0fce42d"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.249389 19372 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260923 23:07:59.250641 19387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.250909 19372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:59.251024 19372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "f0587c36654545f3850069ba68f58d61"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "16da212634d58093162c891d6e85dc7b"
server_key_iv: "56c330984f6f8df01f64a827b0fce42d"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.251133 19372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:59.279448 19372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:59.282270 19372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:59.282464 19372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:59.282727 19372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:59.283077 19372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:59.283135 19372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.283180 19372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:59.283205 19372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.290117 19372 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:35375
I20260923 23:07:59.290177 19500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:35375 every 8 connection(s)
I20260923 23:07:59.290510 19372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:07:59 dist-test-slave-j4c5 krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:59.300693 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.291300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:38659 (local address 127.5.74.62:34511)
0923 23:07:59.291390 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.291393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.291584 (+ 191us) server_negotiation.cc:407] Connection header received
0923 23:07:59.292147 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.292186 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.292200 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.292262 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:59.293257 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.293764 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.294796 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.294997 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.295105 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.297625 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:59.297628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:59.297628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:59.297637 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:59.299130 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.299620 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.299621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.299622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.299688 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.299970 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.299971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.299971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.300096 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:59.300211 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.300496 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.300597 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:07:59.300729 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 19372
I20260923 23:07:59.300876 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:07:59.300918 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.291273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:38659)
0923 23:07:59.291453 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.291499 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.292033 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.292349 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.292354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.292952 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:59.293151 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.293155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.293875 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.293876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.294690 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.294692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.295114 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.295115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.295254 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.295846 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:59.295855 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:59.297517 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:59.299254 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:59.299263 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:59.299266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:59.299523 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:59.299765 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:59.299766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:59.299766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:59.299893 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:59.300198 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:59.300205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:59.300359 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.300475 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.300646 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260923 23:07:59.301123 5416 external_mini_cluster.cc:1428] Setting key 3cf00b0c1effaab93c06a33744aff651
I20260923 23:07:59.301831 19501 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34511
I20260923 23:07:59.301987 19501 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:59.302206 19501 heartbeater.cc:511] Master 127.5.74.62:34511 requested a full tablet report, sending...
I20260923 23:07:59.302842 19176 ts_manager.cc:194] Registered new tserver with Master: f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375)
I20260923 23:07:59.303241 19176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:38659
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:07:59.333896 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34511
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:59.423624 19506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:59.423861 19506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:59.423889 19506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:59.425173 19506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:59.425235 19506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:59.425263 19506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:07:59.426851 19506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:59.427160 19506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:59.427443 19506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:59.432883 19511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:59.432914 19506 server_base.cc:1034] running on GCE node
W20260923 23:07:59.432901 19512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:59.433058 19514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:59.433295 19506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:59.433591 19506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:59.434757 19506 hybrid_clock.cc:648] HybridClock initialized: now 1790204879434733 us; error 45 us; skew 500 ppm
Sep 23 23:07:59 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:59.442031 19506 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:07:59.442484 19506 webserver.cc:460] Webserver started at http://127.5.74.3:42927/ using document root <none> and password file <none>
I20260923 23:07:59.442683 19506 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:59.442725 19506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:59.442824 19506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:07:59.443929 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "80de531c397140b5ac3284c31dcb6e53"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "a550e0c60527ccd47a73ae9659da2cb7"
server_key_iv: "f3226b5bdfa2c5df0c6218d25855261f"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.444319 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "80de531c397140b5ac3284c31dcb6e53"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "a550e0c60527ccd47a73ae9659da2cb7"
server_key_iv: "f3226b5bdfa2c5df0c6218d25855261f"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.445691 19506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:07:59.446554 19521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.446762 19506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:07:59.446861 19506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "80de531c397140b5ac3284c31dcb6e53"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "a550e0c60527ccd47a73ae9659da2cb7"
server_key_iv: "f3226b5bdfa2c5df0c6218d25855261f"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.446921 19506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:59.468763 19506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:59.471158 19506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:59.471294 19506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:59.471527 19506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:59.471881 19506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:07:59.471921 19506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.471944 19506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:07:59.471956 19506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.478013 19506 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:40903
I20260923 23:07:59.478061 19634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:40903 every 8 connection(s)
I20260923 23:07:59.478415 19506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:07:59.480113 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 19506
I20260923 23:07:59.480229 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:07:59.480448 5416 external_mini_cluster.cc:1428] Setting key 8f7acaec2f0de6fe505984bc73f0069d
Sep 23 23:07:59 dist-test-slave-j4c5 krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:59.489553 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.479190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:41073 (local address 127.5.74.62:34511)
0923 23:07:59.479253 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.479255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.479464 (+ 209us) server_negotiation.cc:407] Connection header received
0923 23:07:59.480057 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.480058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.480070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.480130 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:59.481433 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.482088 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.483174 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.483377 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.483447 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.486126 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:59.486127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:59.486127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:59.486135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:59.487823 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.488354 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.488355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.488355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.488412 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.488694 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.488695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.488695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.488820 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:59.488896 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.489250 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.489455 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:07:59.489768 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.479135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:41073)
0923 23:07:59.479286 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.479374 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.479901 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.480248 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.480253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.481076 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:59.481350 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.481355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.482180 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.482181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.483074 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.483076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.483482 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.483483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.483608 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.484222 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:59.484233 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:59.486024 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:59.487913 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:59.487921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:59.487924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:59.488229 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:59.488482 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:59.488483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:59.488483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:59.488603 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:59.488885 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:59.488892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:59.489093 (+ 201us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.489213 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.489509 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260923 23:07:59.490298 19635 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34511
I20260923 23:07:59.490430 19635 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:59.490620 19635 heartbeater.cc:511] Master 127.5.74.62:34511 requested a full tablet report, sending...
I20260923 23:07:59.491147 19176 ts_manager.cc:194] Registered new tserver with Master: 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903)
I20260923 23:07:59.491571 19176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:41073
I20260923 23:07:59.492784 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:07:59.500568 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.494075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.5.74.62:34511)
0923 23:07:59.494122 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.494124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.494213 (+ 89us) server_negotiation.cc:407] Connection header received
0923 23:07:59.494273 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.494274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.494282 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.494335 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:59.494887 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.495401 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.496223 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.496470 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.496533 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.497759 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:59.497761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:59.497761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:59.497769 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:59.499343 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.499712 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.499713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.499713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.499805 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.499996 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.499996 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.499996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.500149 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:59.500311 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.500433 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.500500 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:07:59.502601 19176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:07:59.504285 19176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:07:59.511796 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.507065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:40903 (local address 127.0.0.1:60086)
0923 23:07:59.507463 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.507487 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.507560 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.507878 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.507881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.507884 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:59.507985 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.507987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.509497 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.509498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.510601 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.510604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.511499 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.511499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.511561 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.511573 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.511636 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.511681 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":64,"thread_start_us":56,"threads_started":1}
I20260923 23:07:59.511796 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.507216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.5.74.3:40903)
0923 23:07:59.507723 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.507727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.507736 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:07:59.507772 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.507774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.507790 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.507872 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:59.508073 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.509381 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.510701 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.511395 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.511493 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.511565 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.511651 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.511698 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":152,"threads_started":1}
I20260923 23:07:59.512713 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.507290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:33655 (local address 127.0.0.1:42698)
0923 23:07:59.507970 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.507981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.508029 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.508265 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.508267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.508269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:59.508325 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.508327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.509235 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.509236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.511122 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.511125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.512440 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.512441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.512511 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.512516 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.512583 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.512630 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":318,"thread_start_us":76,"threads_started":1}
I20260923 23:07:59.512986 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.507456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.5.74.1:33655)
0923 23:07:59.507641 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.507646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.508086 (+ 440us) server_negotiation.cc:407] Connection header received
0923 23:07:59.508134 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.508136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.508147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.508253 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:59.508405 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.509129 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.511235 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.512282 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.512367 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.512432 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.512842 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.512889 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260923 23:07:59.512961 19569 tablet_service.cc:1467] Processing CreateTablet for tablet 17ee845aeb0448faaf00d1434c4d132b (DEFAULT_TABLE table=test-table [id=f42b41ad135440698fd3f42924122ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:07:59.513347 19569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ee845aeb0448faaf00d1434c4d132b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:59.513688 19301 tablet_service.cc:1467] Processing CreateTablet for tablet 17ee845aeb0448faaf00d1434c4d132b (DEFAULT_TABLE table=test-table [id=f42b41ad135440698fd3f42924122ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:07:59.513823 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.507034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35375 (local address 127.0.0.1:47476)
0923 23:07:59.507443 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.507488 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.507557 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.507921 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.507922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.507923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:59.507985 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.507987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.509534 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.509535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.511122 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.511125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.513543 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.513544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.513608 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.513611 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.513693 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.513739 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":69,"spinlock_wait_cycles":384,"thread_start_us":115,"threads_started":1}
I20260923 23:07:59.514055 19301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ee845aeb0448faaf00d1434c4d132b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:59.514120 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.507173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.5.74.2:35375)
0923 23:07:59.507660 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.507664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.507674 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:07:59.507737 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.507739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.507755 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.507838 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:59.508069 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.509351 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.512493 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.513364 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.513434 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.513507 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.513710 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.513753 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":153,"threads_started":1}
I20260923 23:07:59.514788 19435 tablet_service.cc:1467] Processing CreateTablet for tablet 17ee845aeb0448faaf00d1434c4d132b (DEFAULT_TABLE table=test-table [id=f42b41ad135440698fd3f42924122ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:07:59.515151 19435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ee845aeb0448faaf00d1434c4d132b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:07:59.517014 19652 tablet_bootstrap.cc:492] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: Bootstrap starting.
I20260923 23:07:59.517367 19653 tablet_bootstrap.cc:492] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Bootstrap starting.
I20260923 23:07:59.517907 19652 tablet_bootstrap.cc:654] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:59.517930 19653 tablet_bootstrap.cc:654] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:59.518425 19653 log.cc:824] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:59.518546 19652 log.cc:824] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:59.519093 19654 tablet_bootstrap.cc:492] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: Bootstrap starting.
I20260923 23:07:59.519286 19653 tablet_bootstrap.cc:492] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: No bootstrap required, opened a new log
I20260923 23:07:59.519369 19653 ts_tablet_manager.cc:1397] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:07:59.519520 19652 tablet_bootstrap.cc:492] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: No bootstrap required, opened a new log
I20260923 23:07:59.519603 19652 ts_tablet_manager.cc:1397] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:07:59.519804 19654 tablet_bootstrap.cc:654] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: Neither blocks nor log segments found. Creating new log.
I20260923 23:07:59.520423 19654 log.cc:824] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:59.521345 19654 tablet_bootstrap.cc:492] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: No bootstrap required, opened a new log
I20260923 23:07:59.521421 19654 ts_tablet_manager.cc:1397] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:07:59.521425 19652 raft_consensus.cc:348] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.521631 19652 raft_consensus.cc:374] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:59.521674 19652 raft_consensus.cc:729] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80de531c397140b5ac3284c31dcb6e53, State: Initialized, Role: FOLLOWER
I20260923 23:07:59.521795 19652 consensus_queue.cc:260] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.521865 19653 raft_consensus.cc:348] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.522053 19653 raft_consensus.cc:374] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:59.522073 19652 ts_tablet_manager.cc:1428] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 23:07:59.522091 19653 raft_consensus.cc:729] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f583019a264222a3479f05968603c1, State: Initialized, Role: FOLLOWER
I20260923 23:07:59.522197 19635 heartbeater.cc:503] Master 127.5.74.62:34511 was elected leader, sending a full tablet report...
I20260923 23:07:59.522224 19653 consensus_queue.cc:260] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.522569 19653 ts_tablet_manager.cc:1428] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:07:59.522830 19367 heartbeater.cc:503] Master 127.5.74.62:34511 was elected leader, sending a full tablet report...
I20260923 23:07:59.523543 19654 raft_consensus.cc:348] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.523705 19654 raft_consensus.cc:374] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:07:59.523748 19654 raft_consensus.cc:729] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0587c36654545f3850069ba68f58d61, State: Initialized, Role: FOLLOWER
I20260923 23:07:59.523870 19654 consensus_queue.cc:260] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.524360 19654 ts_tablet_manager.cc:1428] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 23:07:59.524422 19501 heartbeater.cc:503] Master 127.5.74.62:34511 was elected leader, sending a full tablet report...
W20260923 23:07:59.541129 19502 tablet.cc:2369] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:07:59.570410 19658 raft_consensus.cc:482] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:07:59.570590 19658 raft_consensus.cc:504] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.570989 19658 leader_election.cc:288] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375), b0f583019a264222a3479f05968603c1 (127.5.74.1:33655)
I20260923 23:07:59.574441 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.571606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:38425 (local address 127.5.74.1:33655)
0923 23:07:59.571658 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.571660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.571671 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:07:59.571701 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.571702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.571714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.571765 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:59.571946 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.572532 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.573643 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.574062 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.574152 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.574162 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.574319 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.574358 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:07:59.574465 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.571429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:33655 (local address 127.5.74.3:38425)
0923 23:07:59.571601 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.571609 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.571651 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.571793 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.571794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.571795 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:59.571867 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.571868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.572624 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.572626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.573538 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.573541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.574163 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.574164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.574238 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.574242 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.574314 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.574358 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":31,"thread_start_us":121,"threads_started":1}
I20260923 23:07:59.574551 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.571360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:38831 (local address 127.5.74.2:35375)
0923 23:07:59.571468 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.571470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.571479 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:07:59.571503 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.571504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.571515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.571606 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:59.571998 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.572540 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.573784 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.574176 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.574261 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.574272 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.574439 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.574480 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 23:07:59.574553 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.571229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35375 (local address 127.5.74.3:38831)
0923 23:07:59.571356 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.571369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.571415 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.571665 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.571667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.571669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:59.571737 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.571739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.572645 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.572646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.573653 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.573655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.574273 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.574274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.574355 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.574358 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.574420 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.574462 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":216}
I20260923 23:07:59.574926 19321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ee845aeb0448faaf00d1434c4d132b" candidate_uuid: "80de531c397140b5ac3284c31dcb6e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f583019a264222a3479f05968603c1" is_pre_election: true
I20260923 23:07:59.575024 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ee845aeb0448faaf00d1434c4d132b" candidate_uuid: "80de531c397140b5ac3284c31dcb6e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0587c36654545f3850069ba68f58d61" is_pre_election: true
I20260923 23:07:59.575106 19321 raft_consensus.cc:2454] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80de531c397140b5ac3284c31dcb6e53 in term 0.
I20260923 23:07:59.575186 19455 raft_consensus.cc:2454] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80de531c397140b5ac3284c31dcb6e53 in term 0.
I20260923 23:07:59.575392 19525 leader_election.cc:302] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80de531c397140b5ac3284c31dcb6e53, b0f583019a264222a3479f05968603c1; no voters:
I20260923 23:07:59.575521 19658 raft_consensus.cc:2781] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:07:59.575595 19658 raft_consensus.cc:482] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:07:59.575623 19658 raft_consensus.cc:3037] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:59.576470 19658 raft_consensus.cc:504] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.576623 19658 leader_election.cc:288] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [CANDIDATE]: Term 1 election: Requested vote from peers f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375), b0f583019a264222a3479f05968603c1 (127.5.74.1:33655)
I20260923 23:07:59.576833 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ee845aeb0448faaf00d1434c4d132b" candidate_uuid: "80de531c397140b5ac3284c31dcb6e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0587c36654545f3850069ba68f58d61"
I20260923 23:07:59.576928 19321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ee845aeb0448faaf00d1434c4d132b" candidate_uuid: "80de531c397140b5ac3284c31dcb6e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f583019a264222a3479f05968603c1"
I20260923 23:07:59.576941 19455 raft_consensus.cc:3037] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:59.577018 19321 raft_consensus.cc:3037] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:07:59.577646 19455 raft_consensus.cc:2454] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80de531c397140b5ac3284c31dcb6e53 in term 1.
I20260923 23:07:59.577797 19321 raft_consensus.cc:2454] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80de531c397140b5ac3284c31dcb6e53 in term 1.
I20260923 23:07:59.577813 19525 leader_election.cc:302] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80de531c397140b5ac3284c31dcb6e53, f0587c36654545f3850069ba68f58d61; no voters:
I20260923 23:07:59.577903 19658 raft_consensus.cc:2781] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:07:59.578078 19658 raft_consensus.cc:686] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 1 LEADER]: Becoming Leader. State: Replica: 80de531c397140b5ac3284c31dcb6e53, State: Running, Role: LEADER
I20260923 23:07:59.578192 19658 consensus_queue.cc:237] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.579238 19176 catalog_manager.cc:5495] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3). New cstate: current_term: 1 leader_uuid: "80de531c397140b5ac3284c31dcb6e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } health_report { overall_health: UNKNOWN } } }
I20260923 23:07:59.586670 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.583860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60098 (local address 127.5.74.3:40903)
0923 23:07:59.583963 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.583965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.583975 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:07:59.584008 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.584009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.584029 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.584274 (+ 245us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:07:59.584493 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.584997 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.585846 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.586038 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.586125 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.586313 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:07:59.586434 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:07:59.586534 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.586553 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.586577 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 23:07:59.588848 19321 raft_consensus.cc:1261] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Refusing update from remote peer 80de531c397140b5ac3284c31dcb6e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:07:59.589293 19455 raft_consensus.cc:1261] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [term 1 FOLLOWER]: Refusing update from remote peer 80de531c397140b5ac3284c31dcb6e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:07:59.589401 19658 consensus_queue.cc:1035] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:07:59.589627 19662 consensus_queue.cc:1035] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:07:59.593191 19666 mvcc.cc:205] Tried to move back new op lower bound from 7332679186792275968 to 7332679186753110016. Current Snapshot: MvccSnapshot[applied={T|T < 7332679186792275968}]
I20260923 23:07:59.596282 19175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:07:59.596421 19175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:07:59.597517 19175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Sending DeleteTablet for 3 replicas of tablet 17ee845aeb0448faaf00d1434c4d132b
I20260923 23:07:59.598170 19569 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ee845aeb0448faaf00d1434c4d132b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:59 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:60086
I20260923 23:07:59.598492 19301 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ee845aeb0448faaf00d1434c4d132b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:59 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:42698
I20260923 23:07:59.598506 19435 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ee845aeb0448faaf00d1434c4d132b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:59 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:47476
I20260923 23:07:59.598735 19675 tablet_replica.cc:330] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: stopping tablet replica
I20260923 23:07:59.598857 19675 raft_consensus.cc:2229] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:07:59.598883 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 19238
I20260923 23:07:59.599004 19675 raft_consensus.cc:2258] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:07:59.598870 19677 tablet_replica.cc:330] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: stopping tablet replica
I20260923 23:07:59.599872 19677 raft_consensus.cc:2229] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:07:59.600054 19677 raft_consensus.cc:2258] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:07:59.600128 19675 ts_tablet_manager.cc:1905] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:07:59.601763 19677 ts_tablet_manager.cc:1905] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:07:59.602854 19675 ts_tablet_manager.cc:1918] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:07:59.602947 19675 log.cc:1196] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/17ee845aeb0448faaf00d1434c4d132b
I20260923 23:07:59.603446 19675 ts_tablet_manager.cc:1939] T 17ee845aeb0448faaf00d1434c4d132b P 80de531c397140b5ac3284c31dcb6e53: Deleting consensus metadata
I20260923 23:07:59.603595 19677 ts_tablet_manager.cc:1918] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:07:59.603713 19677 log.cc:1196] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/17ee845aeb0448faaf00d1434c4d132b
I20260923 23:07:59.604029 19161 catalog_manager.cc:4841] TS 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903): tablet 17ee845aeb0448faaf00d1434c4d132b (table test-table [id=f42b41ad135440698fd3f42924122ac7]) successfully deleted
I20260923 23:07:59.604161 19677 ts_tablet_manager.cc:1939] T 17ee845aeb0448faaf00d1434c4d132b P f0587c36654545f3850069ba68f58d61: Deleting consensus metadata
I20260923 23:07:59.604645 19163 catalog_manager.cc:4841] TS f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375): tablet 17ee845aeb0448faaf00d1434c4d132b (table test-table [id=f42b41ad135440698fd3f42924122ac7]) successfully deleted
W20260923 23:07:59.612141 19163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:33655 (error 108)
I20260923 23:07:59.612567 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 19372
I20260923 23:07:59.613097 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.612829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:33655 (local address 127.0.0.1:42702)
0923 23:07:59.613000 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.613052 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:33655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260923 23:07:59.613242 19163 catalog_manager.cc:4584] TS b0f583019a264222a3479f05968603c1 (127.5.74.1:33655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17ee845aeb0448faaf00d1434c4d132b: Network error: Client connection negotiation failed: client connection to 127.5.74.1:33655: connect: Connection refused (error 111)
I20260923 23:07:59.618328 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 19506
I20260923 23:07:59.623934 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 19144
I20260923 23:07:59.629933 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34511
--webserver_interface=127.5.74.62
--webserver_port=33721
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:07:59.716383 19679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:59.716607 19679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:59.716631 19679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:59.717821 19679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:07:59.717864 19679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:59.717880 19679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:59.717893 19679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:07:59.717905 19679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:07:59.719367 19679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:34511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:59.719584 19679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:59.719762 19679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:59.729012 19685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:59.729022 19686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:59.729063 19688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:59.729374 19679 server_base.cc:1034] running on GCE node
I20260923 23:07:59.729665 19679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:59.730029 19679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:59.731231 19679 hybrid_clock.cc:648] HybridClock initialized: now 1790204879731216 us; error 30 us; skew 500 ppm
Sep 23 23:07:59 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:59.734586 19679 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:07:59.735178 19679 webserver.cc:460] Webserver started at http://127.5.74.62:33721/ using document root <none> and password file <none>
I20260923 23:07:59.735436 19679 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:59.735525 19679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:59.737082 19679 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:59.737839 19695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.738114 19679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:59.738229 19679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "0a49c2770989485bb6be45d723bfa657"
format_stamp: "Formatted at 2026-09-23 23:07:58 on dist-test-slave-j4c5"
server_key: "3c30f4757030aa1a13ecddd0095a8cae"
server_key_iv: "1310892f7c2be5b525f3515688a67ffd"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.738674 19679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:59.762979 19679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:59.765724 19679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:59.765921 19679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:59.770475 19679 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:34511
I20260923 23:07:59.770550 19747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:34511 every 8 connection(s)
I20260923 23:07:59.770982 19679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:07:59.772964 19748 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:07:59.773685 19748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Bootstrap starting.
I20260923 23:07:59.776000 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 19679
I20260923 23:07:59.776738 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:33655
--local_ip_for_outbound_sockets=127.5.74.1
--tserver_master_addrs=127.5.74.62:34511
--webserver_port=39545
--webserver_interface=127.5.74.1
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:07:59.782907 19748 log.cc:824] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:59.785070 19748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:07:59.785399 19748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Bootstrap complete.
I20260923 23:07:59.787824 19748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:07:59.788091 19748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a49c2770989485bb6be45d723bfa657, State: Initialized, Role: FOLLOWER
I20260923 23:07:59.788317 19748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:07:59.788404 19748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:07:59.788463 19748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:07:59.788516 19748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:07:59.789602 19748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:07:59.789732 19748 leader_election.cc:302] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a49c2770989485bb6be45d723bfa657; no voters:
I20260923 23:07:59.789927 19748 leader_election.cc:288] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 23:07:59.790007 19752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:07:59.790172 19752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 2 LEADER]: Becoming Leader. State: Replica: 0a49c2770989485bb6be45d723bfa657, State: Running, Role: LEADER
I20260923 23:07:59.790213 19748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:07:59.790381 19752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:07:59.790668 19752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a49c2770989485bb6be45d723bfa657. Latest consensus state: current_term: 2 leader_uuid: "0a49c2770989485bb6be45d723bfa657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } } }
I20260923 23:07:59.790797 19752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:59.790669 19753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0a49c2770989485bb6be45d723bfa657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } } }
I20260923 23:07:59.791129 19753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: This master's current role is: LEADER
I20260923 23:07:59.791260 19761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:07:59.792080 19761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:07:59.792212 19761 catalog_manager.cc:1259] Loaded cluster ID: e605a38d855b4a4b9177a3d180057a2a
I20260923 23:07:59.792263 19761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:07:59.792841 19761 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 23:07:59.797957 19761 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 23:07:59.800760 19761 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:07:59.800997 19761 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Loaded TSK: 0
I20260923 23:07:59.801203 19761 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:07:59.865449 19750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:07:59.865672 19750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:07:59.865700 19750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:07:59.866945 19750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:07:59.866997 19750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:07:59.867026 19750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:07:59.868574 19750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:33655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=39545
--tserver_master_addrs=127.5.74.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:07:59.868840 19750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:07:59.869099 19750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:07:59.874729 19775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:59.874729 19776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:07:59.874733 19778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:07:59.875330 19750 server_base.cc:1034] running on GCE node
I20260923 23:07:59.875556 19750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:07:59.875859 19750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:07:59.877049 19750 hybrid_clock.cc:648] HybridClock initialized: now 1790204879877034 us; error 26 us; skew 500 ppm
Sep 23 23:07:59 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:07:59.880268 19750 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:07:59.880803 19750 webserver.cc:460] Webserver started at http://127.5.74.1:39545/ using document root <none> and password file <none>
I20260923 23:07:59.881052 19750 fs_manager.cc:359] Metadata directory not provided
I20260923 23:07:59.881135 19750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:07:59.882632 19750 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:07:59.883455 19785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:07:59.883683 19750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:07:59.883800 19750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "b0f583019a264222a3479f05968603c1"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "b854b8ae16eb4b39ef2d6df0c9be799f"
server_key_iv: "e40db2cbc8a046f7f19f5afa7930b159"
server_key_version: "encryptionkey@0"
I20260923 23:07:59.884227 19750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:07:59.899547 19750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:07:59.902482 19750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:07:59.902675 19750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:07:59.902963 19750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:07:59.903510 19792 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:07:59.905467 19750 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:07:59.905529 19750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:07:59.905557 19750 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:07:59.906257 19750 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:07:59.906315 19750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 23:07:59.906350 19792 tablet_bootstrap.cc:492] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Bootstrap starting.
I20260923 23:07:59.914513 19750 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:33655
I20260923 23:07:59.914732 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:33655 every 8 connection(s)
I20260923 23:07:59.915081 19750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:07:59.918452 19792 log.cc:824] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:07:59.920885 19792 tablet_bootstrap.cc:492] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:07:59.921208 19792 tablet_bootstrap.cc:492] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Bootstrap complete.
I20260923 23:07:59.922281 19792 ts_tablet_manager.cc:1397] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.002s
I20260923 23:07:59.922712 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 19750
I20260923 23:07:59.923148 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:35375
--local_ip_for_outbound_sockets=127.5.74.2
--tserver_master_addrs=127.5.74.62:34511
--webserver_port=46275
--webserver_interface=127.5.74.2
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:07:59.924314 19792 raft_consensus.cc:348] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.924654 19792 raft_consensus.cc:729] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f583019a264222a3479f05968603c1, State: Initialized, Role: FOLLOWER
Sep 23 23:07:59 dist-test-slave-j4c5 krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:07:59.924834 19792 consensus_queue.cc:260] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:07:59.925133 19792 ts_tablet_manager.cc:1428] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:07:59.930632 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.915829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:42423)
0923 23:07:59.916269 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.916334 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.917175 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.917734 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.917738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.918159 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:07:59.918461 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.918468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.920214 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.920215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.921521 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.921524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.922366 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.922367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.922523 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.923180 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:07:59.923201 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:07:59.925372 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:07:59.928088 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:59.928101 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:59.928104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:59.928567 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:59.928977 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:07:59.928978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:07:59.928978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:07:59.929127 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:07:59.929952 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:07:59.929967 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:07:59.930128 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.930233 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.930416 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":96,"threads_started":1}
I20260923 23:07:59.931660 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.915967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:42423 (local address 127.5.74.62:34511)
0923 23:07:59.916220 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.916231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.916468 (+ 237us) server_negotiation.cc:407] Connection header received
0923 23:07:59.917437 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.917444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.917504 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.917646 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:07:59.918647 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.920039 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.921630 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.922257 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.922370 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.925522 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:07:59.925528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:07:59.925528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:07:59.925543 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:07:59.927961 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.928698 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.928702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.928702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.928831 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:07:59.929337 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:07:59.929339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:07:59.929339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:07:59.929851 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:07:59.929958 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.930348 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.930516 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260923 23:07:59.932405 19900 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34511
I20260923 23:07:59.932514 19900 heartbeater.cc:464] Registering TS with master...
I20260923 23:07:59.932821 19900 heartbeater.cc:511] Master 127.5.74.62:34511 requested a full tablet report, sending...
I20260923 23:07:59.933758 19712 ts_manager.cc:194] Registered new tserver with Master: b0f583019a264222a3479f05968603c1 (127.5.74.1:33655)
I20260923 23:07:59.935070 19712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:42423
I20260923 23:07:59.936185 19900 heartbeater.cc:503] Master 127.5.74.62:34511 was elected leader, sending a full tablet report...
I20260923 23:07:59.940929 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.934496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:33655 (local address 127.0.0.1:42708)
0923 23:07:59.934703 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 23:07:59.934754 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 23:07:59.934974 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:07:59.936703 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:07:59.936706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:07:59.936709 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:07:59.936796 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.936799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.938206 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.938207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.939599 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:07:59.939603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.940606 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.940607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:07:59.940673 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.940684 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 23:07:59.940755 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 23:07:59.940853 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":97,"threads_started":1}
I20260923 23:07:59.940981 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:07:59.934713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.5.74.1:33655)
0923 23:07:59.936445 (+ 1732us) server_negotiation.cc:206] Beginning negotiation
0923 23:07:59.936450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:07:59.936461 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:07:59.936503 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:07:59.936505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:07:59.936522 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:07:59.936886 (+ 364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:07:59.936942 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.938101 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.939692 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:07:59.940506 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:07:59.940585 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:07:59.940644 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:07:59.940817 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 23:07:59.940903 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1544,"thread_start_us":70,"threads_started":1}
I20260923 23:07:59.941768 19834 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ee845aeb0448faaf00d1434c4d132b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:07:59 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:42708
I20260923 23:07:59.942516 19909 tablet_replica.cc:330] stopping tablet replica
I20260923 23:07:59.942636 19909 raft_consensus.cc:2229] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:07:59.942728 19909 raft_consensus.cc:2258] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:07:59.943106 19909 ts_tablet_manager.cc:1905] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:07:59.945633 19909 ts_tablet_manager.cc:1918] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:07:59.945713 19909 log.cc:1196] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/17ee845aeb0448faaf00d1434c4d132b
I20260923 23:07:59.946079 19909 ts_tablet_manager.cc:1939] T 17ee845aeb0448faaf00d1434c4d132b P b0f583019a264222a3479f05968603c1: Deleting consensus metadata
I20260923 23:07:59.947237 19699 catalog_manager.cc:4841] TS b0f583019a264222a3479f05968603c1 (127.5.74.1:33655): tablet 17ee845aeb0448faaf00d1434c4d132b (table test-table [id=f42b41ad135440698fd3f42924122ac7]) successfully deleted
W20260923 23:08:00.021344 19905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:00.021624 19905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:00.021930 19905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:00.023940 19905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:00.024049 19905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:00.024109 19905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:08:00.025894 19905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:35375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=46275
--tserver_master_addrs=127.5.74.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:00.026206 19905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:00.026473 19905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:00.032049 19914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:00.032262 19917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:00.032307 19915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:00.033037 19905 server_base.cc:1034] running on GCE node
I20260923 23:08:00.033298 19905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:00.033592 19905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:00.034840 19905 hybrid_clock.cc:648] HybridClock initialized: now 1790204880034813 us; error 55 us; skew 500 ppm
Sep 23 23:08:00 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204880, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:00.037791 19905 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:08:00.038466 19905 webserver.cc:460] Webserver started at http://127.5.74.2:46275/ using document root <none> and password file <none>
I20260923 23:08:00.038738 19905 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:00.038802 19905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:00.040378 19905 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:00.041237 19924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.041594 19905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:00.041680 19905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "f0587c36654545f3850069ba68f58d61"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "16da212634d58093162c891d6e85dc7b"
server_key_iv: "56c330984f6f8df01f64a827b0fce42d"
server_key_version: "encryptionkey@0"
I20260923 23:08:00.042160 19905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:00.063858 19905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:00.066380 19905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:00.066509 19905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:00.066710 19905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:00.067057 19905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:00.067094 19905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.067118 19905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:00.067131 19905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.073665 19905 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:35375
I20260923 23:08:00.073783 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:35375 every 8 connection(s)
I20260923 23:08:00.074118 19905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:08:00 dist-test-slave-j4c5 krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204880, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:08:00.081924 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 19905
I20260923 23:08:00.082305 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:40903
--local_ip_for_outbound_sockets=127.5.74.3
--tserver_master_addrs=127.5.74.62:34511
--webserver_port=42927
--webserver_interface=127.5.74.3
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:08:00.087802 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.075003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:41239 (local address 127.5.74.62:34511)
0923 23:08:00.075076 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:00.075078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:00.075235 (+ 157us) server_negotiation.cc:407] Connection header received
0923 23:08:00.075767 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:00.075769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:00.075781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:00.075830 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:00.076921 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.077448 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.078473 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.078676 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.078761 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.081253 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:00.081255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:00.081255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:00.081263 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:00.084950 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:00.085716 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:00.085717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:00.085718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:00.085782 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:00.086351 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:00.086352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:00.086353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:00.086538 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:00.086602 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:00.087600 (+ 998us) server_negotiation.cc:299] Negotiation successful
0923 23:08:00.087708 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 23:08:00.088094 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.074917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:41239)
0923 23:08:00.075087 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.075130 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:00.075654 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:00.075912 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:00.075916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:00.076608 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:00.076813 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.076817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.077548 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.077549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.078362 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.078364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.078782 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.078783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.078915 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.079432 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:00.079442 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:00.081108 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:00.085071 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:00.085207 (+ 136us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:00.085214 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:00.085609 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:00.085892 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:00.085893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:00.085893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:00.086243 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:00.086618 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:00.086626 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:00.086825 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 23:08:00.087476 (+ 651us) client_negotiation.cc:240] Negotiation successful
0923 23:08:00.087724 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260923 23:08:00.088948 20038 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34511
I20260923 23:08:00.089051 20038 heartbeater.cc:464] Registering TS with master...
I20260923 23:08:00.089253 20038 heartbeater.cc:511] Master 127.5.74.62:34511 requested a full tablet report, sending...
I20260923 23:08:00.089751 19712 ts_manager.cc:194] Registered new tserver with Master: f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375)
I20260923 23:08:00.090778 19712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:41239
W20260923 23:08:00.174216 20041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:00.174453 20041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:00.174479 20041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:00.175645 20041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:00.175684 20041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:00.175711 20041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:08:00.177240 20041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:40903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=42927
--tserver_master_addrs=127.5.74.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:00.177453 20041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:00.177635 20041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:00.182796 20046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:00.183004 20047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:00.183027 20049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:00.182999 20041 server_base.cc:1034] running on GCE node
I20260923 23:08:00.183350 20041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:00.183614 20041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:00.184787 20041 hybrid_clock.cc:648] HybridClock initialized: now 1790204880184761 us; error 43 us; skew 500 ppm
Sep 23 23:08:00 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204880, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:00.187433 20041 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:08:00.187995 20041 webserver.cc:460] Webserver started at http://127.5.74.3:42927/ using document root <none> and password file <none>
I20260923 23:08:00.188314 20041 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:00.188400 20041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:00.189864 20041 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:00.190629 20056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.190860 20041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:00.190963 20041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "80de531c397140b5ac3284c31dcb6e53"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "a550e0c60527ccd47a73ae9659da2cb7"
server_key_iv: "f3226b5bdfa2c5df0c6218d25855261f"
server_key_version: "encryptionkey@0"
I20260923 23:08:00.191318 20041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:00.213338 20041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:00.215765 20041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:00.216078 20041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:00.216403 20041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:00.216735 20041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:00.216789 20041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.216822 20041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:00.216854 20041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.223619 20041 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:40903
I20260923 23:08:00.223663 20169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:40903 every 8 connection(s)
I20260923 23:08:00.224085 20041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:08:00.229810 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 20041
Sep 23 23:08:00 dist-test-slave-j4c5 krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204880, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:08:00.235697 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.225061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:49609 (local address 127.5.74.62:34511)
0923 23:08:00.225151 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:00.225154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:00.225342 (+ 188us) server_negotiation.cc:407] Connection header received
0923 23:08:00.225944 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:00.225945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:00.225960 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:00.226024 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:00.227040 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.227574 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.228776 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.228991 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.229070 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.232073 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:00.232075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:00.232075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:00.232084 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:00.233824 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:00.234536 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:00.234537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:00.234537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:00.234597 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:00.234948 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:00.234950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:00.234950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:00.235072 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:00.235181 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:00.235490 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 23:08:00.235598 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 23:08:00.235915 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.225030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:49609)
0923 23:08:00.225213 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.225251 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:00.225799 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:00.226106 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:00.226109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:00.226715 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:00.226917 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.226922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.227789 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.227790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.228666 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.228668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.229078 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.229078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.229212 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.230131 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:00.230140 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:00.231958 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:00.233981 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:00.233989 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:00.233992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:00.234437 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:00.234680 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:00.234681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:00.234682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:00.234842 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:00.235172 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:00.235179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:00.235341 (+ 162us) client_negotiation.cc:769] Sending connection context
0923 23:08:00.235466 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 23:08:00.235652 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260923 23:08:00.236497 20170 heartbeater.cc:347] Connected to a master server at 127.5.74.62:34511
I20260923 23:08:00.236637 20170 heartbeater.cc:464] Registering TS with master...
I20260923 23:08:00.236852 20170 heartbeater.cc:511] Master 127.5.74.62:34511 requested a full tablet report, sending...
I20260923 23:08:00.237322 19712 ts_manager.cc:194] Registered new tserver with Master: 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903)
I20260923 23:08:00.237797 19712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:49609
I20260923 23:08:00.238193 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.230305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.5.74.62:34511)
0923 23:08:00.230427 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:00.230429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:00.230439 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:08:00.230533 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:00.230534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:00.230546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:00.230625 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:00.231425 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.232040 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.233293 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.233548 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.233616 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.235037 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:00.235040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:00.235040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:00.235047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:00.236873 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:00.237311 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:00.237312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:00.237312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:00.237382 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:00.237623 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:00.237624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:00.237625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:00.237737 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:00.237918 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:00.237998 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 23:08:00.238088 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260923 23:08:00.238961 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:08:00.242000 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.239384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.5.74.62:34511)
0923 23:08:00.239432 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:00.239434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:00.239546 (+ 112us) server_negotiation.cc:407] Connection header received
0923 23:08:00.239598 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:00.239598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:00.239627 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:00.239691 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:08:00.239770 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.240310 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.241141 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.241368 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.241435 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.241661 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:08:00.241778 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:08:00.241854 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:00.241924 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 23:08:00.241945 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 23:08:00.242491 19712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:08:00.244127 19712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:08:00.247171 19834 tablet_service.cc:1467] Processing CreateTablet for tablet 301d2a2bbac843e597e151cb90e44ede (DEFAULT_TABLE table=test-table [id=4877edf84cfe408a8c3495a44e2d4ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:08:00.247521 19834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301d2a2bbac843e597e151cb90e44ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:00.248904 19792 tablet_bootstrap.cc:492] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Bootstrap starting.
I20260923 23:08:00.249266 19792 tablet_bootstrap.cc:654] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:00.249838 19792 tablet_bootstrap.cc:492] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: No bootstrap required, opened a new log
I20260923 23:08:00.249907 19792 ts_tablet_manager.cc:1397] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:00.250036 19792 raft_consensus.cc:348] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.250129 19792 raft_consensus.cc:374] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:00.250191 19792 raft_consensus.cc:729] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f583019a264222a3479f05968603c1, State: Initialized, Role: FOLLOWER
I20260923 23:08:00.250242 19792 consensus_queue.cc:260] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.250340 19792 ts_tablet_manager.cc:1428] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:08:00.251780 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.246769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.5.74.3:40903)
0923 23:08:00.247468 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:00.247473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:00.247485 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:08:00.247523 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:00.247525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:00.247542 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:00.247808 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:00.247883 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.248946 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.250675 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.251440 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.251508 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.251555 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:00.251679 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 23:08:00.251712 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":127,"threads_started":1}
I20260923 23:08:00.251806 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.246570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:40903 (local address 127.0.0.1:60104)
0923 23:08:00.246628 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.246639 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:00.246688 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:00.247684 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:00.247685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:00.247687 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:00.247732 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.247733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.249067 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.249068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.250577 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.250582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.251527 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.251528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.251588 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.251594 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 23:08:00.251655 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:08:00.251710 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":35}
I20260923 23:08:00.252707 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.246718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35375 (local address 127.0.0.1:47480)
0923 23:08:00.246867 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.246891 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:00.246999 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:00.247599 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:00.247600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:00.247603 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:00.247657 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.247659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.249204 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.249205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.250933 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.250936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.252430 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.252431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.252501 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.252505 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:08:00.252564 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 23:08:00.252606 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":42,"thread_start_us":78,"threads_started":1}
I20260923 23:08:00.253041 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.247056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.5.74.2:35375)
0923 23:08:00.247316 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:00.247321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:00.247333 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:08:00.247405 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:00.247406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:00.247423 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:00.247509 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:00.247726 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.248921 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.251068 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.251797 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.252241 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.252343 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:00.252938 (+ 595us) server_negotiation.cc:299] Negotiation successful
0923 23:08:00.252979 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":159,"threads_started":1}
I20260923 23:08:00.252944 20104 tablet_service.cc:1467] Processing CreateTablet for tablet 301d2a2bbac843e597e151cb90e44ede (DEFAULT_TABLE table=test-table [id=4877edf84cfe408a8c3495a44e2d4ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:08:00.253322 20104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301d2a2bbac843e597e151cb90e44ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:00.253685 19972 tablet_service.cc:1467] Processing CreateTablet for tablet 301d2a2bbac843e597e151cb90e44ede (DEFAULT_TABLE table=test-table [id=4877edf84cfe408a8c3495a44e2d4ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:08:00.254045 19972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301d2a2bbac843e597e151cb90e44ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:00.256500 20180 tablet_bootstrap.cc:492] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: Bootstrap starting.
I20260923 23:08:00.257360 20181 tablet_bootstrap.cc:492] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: Bootstrap starting.
I20260923 23:08:00.257359 20180 tablet_bootstrap.cc:654] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:00.258019 20181 tablet_bootstrap.cc:654] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:00.258343 20180 log.cc:824] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:00.258626 20181 log.cc:824] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:00.259250 20180 tablet_bootstrap.cc:492] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: No bootstrap required, opened a new log
I20260923 23:08:00.259342 20180 ts_tablet_manager.cc:1397] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:08:00.259418 20181 tablet_bootstrap.cc:492] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: No bootstrap required, opened a new log
I20260923 23:08:00.259505 20181 ts_tablet_manager.cc:1397] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:08:00.260897 20180 raft_consensus.cc:348] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.261081 20180 raft_consensus.cc:374] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:00.260991 20181 raft_consensus.cc:348] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.261130 20180 raft_consensus.cc:729] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80de531c397140b5ac3284c31dcb6e53, State: Initialized, Role: FOLLOWER
I20260923 23:08:00.261168 20181 raft_consensus.cc:374] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:00.261211 20181 raft_consensus.cc:729] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0587c36654545f3850069ba68f58d61, State: Initialized, Role: FOLLOWER
I20260923 23:08:00.261525 20181 consensus_queue.cc:260] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.261515 20180 consensus_queue.cc:260] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.262028 20181 ts_tablet_manager.cc:1428] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 23:08:00.262105 20180 ts_tablet_manager.cc:1428] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:08:00.262279 20038 heartbeater.cc:503] Master 127.5.74.62:34511 was elected leader, sending a full tablet report...
I20260923 23:08:00.262379 20170 heartbeater.cc:503] Master 127.5.74.62:34511 was elected leader, sending a full tablet report...
W20260923 23:08:00.324779 20039 tablet.cc:2369] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:08:00.357936 20185 raft_consensus.cc:482] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:08:00.358094 20185 raft_consensus.cc:504] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.358603 20185 leader_election.cc:288] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375), b0f583019a264222a3479f05968603c1 (127.5.74.1:33655)
I20260923 23:08:00.361964 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.358840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35375 (local address 127.5.74.3:50485)
0923 23:08:00.358904 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.358918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:00.358973 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:00.359213 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:00.359214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:00.359216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:00.359294 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.359296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.360072 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.360073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.361061 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.361064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.361673 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.361674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.361767 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.361771 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:08:00.361838 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:08:00.361890 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":52}
I20260923 23:08:00.362022 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.358992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:50485 (local address 127.5.74.2:35375)
0923 23:08:00.359048 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:00.359051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:00.359062 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:08:00.359094 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:00.359094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:00.359106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:00.359195 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:00.359433 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.359969 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.361162 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.361577 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.361634 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.361644 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:00.361861 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 23:08:00.361935 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:08:00.362133 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.358977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:33655 (local address 127.5.74.3:37585)
0923 23:08:00.359160 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.359170 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:00.359227 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:00.359375 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:00.359375 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:00.359377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:00.359497 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.359499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.360350 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.360351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.361124 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:00.361127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.361865 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.361866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:00.361938 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.361942 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:08:00.362002 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 23:08:00.362044 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260923 23:08:00.362167 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.359065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:37585 (local address 127.5.74.1:33655)
0923 23:08:00.359149 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:00.359150 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:00.359215 (+ 65us) server_negotiation.cc:407] Connection header received
0923 23:08:00.359296 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:00.359297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:00.359309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:00.359367 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:00.359606 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.360210 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.361234 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.361750 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.361838 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.361849 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:00.362028 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 23:08:00.362082 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 23:08:00.362424 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301d2a2bbac843e597e151cb90e44ede" candidate_uuid: "80de531c397140b5ac3284c31dcb6e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0587c36654545f3850069ba68f58d61" is_pre_election: true
I20260923 23:08:00.362412 19854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301d2a2bbac843e597e151cb90e44ede" candidate_uuid: "80de531c397140b5ac3284c31dcb6e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f583019a264222a3479f05968603c1" is_pre_election: true
I20260923 23:08:00.362591 19992 raft_consensus.cc:2454] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80de531c397140b5ac3284c31dcb6e53 in term 0.
I20260923 23:08:00.362591 19854 raft_consensus.cc:2454] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80de531c397140b5ac3284c31dcb6e53 in term 0.
I20260923 23:08:00.362831 20060 leader_election.cc:302] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80de531c397140b5ac3284c31dcb6e53, b0f583019a264222a3479f05968603c1; no voters:
I20260923 23:08:00.363019 20185 raft_consensus.cc:2781] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:08:00.363137 20185 raft_consensus.cc:482] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:08:00.363175 20185 raft_consensus.cc:3037] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:00.364058 20185 raft_consensus.cc:504] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.364259 20185 leader_election.cc:288] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [CANDIDATE]: Term 1 election: Requested vote from peers f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375), b0f583019a264222a3479f05968603c1 (127.5.74.1:33655)
I20260923 23:08:00.364530 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301d2a2bbac843e597e151cb90e44ede" candidate_uuid: "80de531c397140b5ac3284c31dcb6e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0587c36654545f3850069ba68f58d61"
I20260923 23:08:00.364530 19854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301d2a2bbac843e597e151cb90e44ede" candidate_uuid: "80de531c397140b5ac3284c31dcb6e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f583019a264222a3479f05968603c1"
I20260923 23:08:00.364626 19992 raft_consensus.cc:3037] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:00.364625 19854 raft_consensus.cc:3037] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:00.365685 19992 raft_consensus.cc:2454] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80de531c397140b5ac3284c31dcb6e53 in term 1.
I20260923 23:08:00.365927 20060 leader_election.cc:302] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80de531c397140b5ac3284c31dcb6e53, f0587c36654545f3850069ba68f58d61; no voters:
I20260923 23:08:00.366040 20185 raft_consensus.cc:2781] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:08:00.366086 19854 raft_consensus.cc:2454] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80de531c397140b5ac3284c31dcb6e53 in term 1.
I20260923 23:08:00.366231 20185 raft_consensus.cc:686] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 1 LEADER]: Becoming Leader. State: Replica: 80de531c397140b5ac3284c31dcb6e53, State: Running, Role: LEADER
I20260923 23:08:00.366359 20185 consensus_queue.cc:237] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.367362 19712 catalog_manager.cc:5495] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3). New cstate: current_term: 1 leader_uuid: "80de531c397140b5ac3284c31dcb6e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } health_report { overall_health: UNKNOWN } } }
I20260923 23:08:00.402218 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.399404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.5.74.3:40903)
0923 23:08:00.399475 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:00.399478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:00.399487 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:08:00.399537 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:00.399538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:00.399558 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:00.399635 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:08:00.399771 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.400395 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.401191 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:00.401420 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:00.401497 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:00.401666 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:08:00.401801 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:08:00.401867 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:00.402012 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 23:08:00.402074 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 23:08:00.404436 19992 raft_consensus.cc:1261] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [term 1 FOLLOWER]: Refusing update from remote peer 80de531c397140b5ac3284c31dcb6e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:08:00.404436 19854 raft_consensus.cc:1261] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Refusing update from remote peer 80de531c397140b5ac3284c31dcb6e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:08:00.404886 20187 consensus_queue.cc:1035] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:08:00.405040 20185 consensus_queue.cc:1035] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:08:00.407203 20189 mvcc.cc:205] Tried to move back new op lower bound from 7332679190132953088 to 7332679189981429760. Current Snapshot: MvccSnapshot[applied={T|T < 7332679190132953088}]
I20260923 23:08:00.407218 20190 mvcc.cc:205] Tried to move back new op lower bound from 7332679190132953088 to 7332679189981429760. Current Snapshot: MvccSnapshot[applied={T|T < 7332679190132953088}]
I20260923 23:08:00.407935 20191 mvcc.cc:205] Tried to move back new op lower bound from 7332679190132953088 to 7332679189981429760. Current Snapshot: MvccSnapshot[applied={T|T < 7332679190132953088}]
I20260923 23:08:00.409723 19711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:08:00.409833 19711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:08:00.410719 19711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Sending DeleteTablet for 3 replicas of tablet 301d2a2bbac843e597e151cb90e44ede
I20260923 23:08:00.411270 20104 tablet_service.cc:1514] Processing DeleteTablet for tablet 301d2a2bbac843e597e151cb90e44ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:08:00 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:60104
I20260923 23:08:00.411366 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 19750
I20260923 23:08:00.411913 19972 tablet_service.cc:1514] Processing DeleteTablet for tablet 301d2a2bbac843e597e151cb90e44ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:08:00 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:47480
I20260923 23:08:00.412169 20199 tablet_replica.cc:330] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: stopping tablet replica
I20260923 23:08:00.412279 20199 raft_consensus.cc:2229] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:08:00.412398 20199 raft_consensus.cc:2258] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:08:00.412881 20199 ts_tablet_manager.cc:1905] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:08:00.413275 20198 tablet_replica.cc:330] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: stopping tablet replica
I20260923 23:08:00.413416 20198 raft_consensus.cc:2229] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:08:00.413574 20198 raft_consensus.cc:2258] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:08:00.414062 20198 ts_tablet_manager.cc:1905] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:08:00.415139 20199 ts_tablet_manager.cc:1918] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:08:00.415230 20199 log.cc:1196] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/301d2a2bbac843e597e151cb90e44ede
I20260923 23:08:00.415536 20199 ts_tablet_manager.cc:1939] T 301d2a2bbac843e597e151cb90e44ede P f0587c36654545f3850069ba68f58d61: Deleting consensus metadata
I20260923 23:08:00.416062 20198 ts_tablet_manager.cc:1918] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:08:00.416229 20198 log.cc:1196] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/301d2a2bbac843e597e151cb90e44ede
I20260923 23:08:00.416455 19699 catalog_manager.cc:4841] TS f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375): tablet 301d2a2bbac843e597e151cb90e44ede (table test-table [id=4877edf84cfe408a8c3495a44e2d4ebc]) successfully deleted
I20260923 23:08:00.416636 20198 ts_tablet_manager.cc:1939] T 301d2a2bbac843e597e151cb90e44ede P 80de531c397140b5ac3284c31dcb6e53: Deleting consensus metadata
I20260923 23:08:00.417162 19697 catalog_manager.cc:4841] TS 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903): tablet 301d2a2bbac843e597e151cb90e44ede (table test-table [id=4877edf84cfe408a8c3495a44e2d4ebc]) successfully deleted
W20260923 23:08:00.418354 19699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:33655 (error 108)
I20260923 23:08:00.418628 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 19905
I20260923 23:08:00.419232 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.418779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:33655 (local address 127.0.0.1:42724)
0923 23:08:00.419149 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.419194 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:33655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348}
W20260923 23:08:00.419512 19699 catalog_manager.cc:4584] TS b0f583019a264222a3479f05968603c1 (127.5.74.1:33655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 301d2a2bbac843e597e151cb90e44ede: Network error: Client connection negotiation failed: client connection to 127.5.74.1:33655: connect: Connection refused (error 111)
I20260923 23:08:00.425124 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 20041
I20260923 23:08:00.430421 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 19679
I20260923 23:08:00.435918 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34511
--webserver_interface=127.5.74.62
--webserver_port=33721
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:08:00.522998 20201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:00.523273 20201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:00.523311 20201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:00.524624 20201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:08:00.524708 20201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:00.524734 20201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:00.524756 20201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:08:00.524785 20201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:08:00.526335 20201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:34511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:00.526647 20201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:00.526897 20201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:00.532430 20206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:00.532384 20209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:00.532450 20201 server_base.cc:1034] running on GCE node
W20260923 23:08:00.532410 20207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:00.533005 20201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:00.533335 20201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:00.534495 20201 hybrid_clock.cc:648] HybridClock initialized: now 1790204880534479 us; error 38 us; skew 500 ppm
Sep 23 23:08:00 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204880, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:00.537350 20201 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:08:00.537789 20201 webserver.cc:460] Webserver started at http://127.5.74.62:33721/ using document root <none> and password file <none>
I20260923 23:08:00.538041 20201 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:00.538112 20201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:00.539597 20201 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:08:00.540396 20216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.540648 20201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:08:00.540793 20201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "0a49c2770989485bb6be45d723bfa657"
format_stamp: "Formatted at 2026-09-23 23:07:58 on dist-test-slave-j4c5"
server_key: "3c30f4757030aa1a13ecddd0095a8cae"
server_key_iv: "1310892f7c2be5b525f3515688a67ffd"
server_key_version: "encryptionkey@0"
I20260923 23:08:00.541257 20201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:00.560863 20201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:00.563652 20201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:00.563824 20201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:00.568653 20201 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:34511
I20260923 23:08:00.568717 20268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:34511 every 8 connection(s)
I20260923 23:08:00.569105 20201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:08:00.570710 20269 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:08:00.571380 20269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Bootstrap starting.
I20260923 23:08:00.571682 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 20201
I20260923 23:08:00.572028 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:33655
--local_ip_for_outbound_sockets=127.5.74.1
--tserver_master_addrs=127.5.74.62:34511
--webserver_port=39545
--webserver_interface=127.5.74.1
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:08:00.580766 20269 log.cc:824] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:00.583334 20269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:08:00.583701 20269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657: Bootstrap complete.
I20260923 23:08:00.586223 20269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:08:00.586498 20269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a49c2770989485bb6be45d723bfa657, State: Initialized, Role: FOLLOWER
I20260923 23:08:00.586709 20269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:08:00.586812 20269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:08:00.586864 20269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:08:00.586915 20269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 2 FOLLOWER]: Advancing to term 3
I20260923 23:08:00.587940 20269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:08:00.588064 20269 leader_election.cc:302] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a49c2770989485bb6be45d723bfa657; no voters:
I20260923 23:08:00.588282 20269 leader_election.cc:288] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 23:08:00.588322 20273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 23:08:00.588428 20273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [term 3 LEADER]: Becoming Leader. State: Replica: 0a49c2770989485bb6be45d723bfa657, State: Running, Role: LEADER
I20260923 23:08:00.588526 20273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } }
I20260923 23:08:00.588573 20269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:08:00.588799 20274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0a49c2770989485bb6be45d723bfa657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } } }
I20260923 23:08:00.588891 20274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: This master's current role is: LEADER
I20260923 23:08:00.589161 20277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:08:00.589413 20275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a49c2770989485bb6be45d723bfa657. Latest consensus state: current_term: 3 leader_uuid: "0a49c2770989485bb6be45d723bfa657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49c2770989485bb6be45d723bfa657" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 34511 } } }
I20260923 23:08:00.589546 20275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a49c2770989485bb6be45d723bfa657 [sys.catalog]: This master's current role is: LEADER
I20260923 23:08:00.590129 20277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:08:00.590271 20277 catalog_manager.cc:1259] Loaded cluster ID: e605a38d855b4a4b9177a3d180057a2a
I20260923 23:08:00.590519 20277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 23:08:00.590757 20277 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdda6bc29ad google::LogMessage::Fail() at ??:0
@ 0x7fdda6bc588a google::LogMessage::SendToLog() at ??:0
@ 0x7fdda6bc24de google::LogMessage::Flush() at ??:0
@ 0x7fdda6bc3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fddaeae30f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fddaeaec8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdda8a900d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdda8a85954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdda98e96db start_thread at ??:0
@ 0x7fdda56c061f clone at ??:0
W20260923 23:08:00.677851 20271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:00.678067 20271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:00.678092 20271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:00.679365 20271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:00.679421 20271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:00.679451 20271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:08:00.681178 20271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:33655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=39545
--tserver_master_addrs=127.5.74.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:00.681478 20271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:00.681752 20271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:00.687170 20306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:00.687177 20305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:00.687309 20308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:00.687875 20271 server_base.cc:1034] running on GCE node
I20260923 23:08:00.688122 20271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:00.688454 20271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:00.689656 20271 hybrid_clock.cc:648] HybridClock initialized: now 1790204880689630 us; error 45 us; skew 500 ppm
Sep 23 23:08:00 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204880, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:00.692673 20271 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:08:00.693188 20271 webserver.cc:460] Webserver started at http://127.5.74.1:39545/ using document root <none> and password file <none>
I20260923 23:08:00.693464 20271 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:00.693542 20271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:00.695086 20271 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 23:08:00.695952 20315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.696326 20271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:08:00.696440 20271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "b0f583019a264222a3479f05968603c1"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "b854b8ae16eb4b39ef2d6df0c9be799f"
server_key_iv: "e40db2cbc8a046f7f19f5afa7930b159"
server_key_version: "encryptionkey@0"
I20260923 23:08:00.696945 20271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:00.712677 20271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:00.715428 20271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:00.715598 20271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:00.715893 20271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:00.716459 20322 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:08:00.718055 20271 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:08:00.718119 20271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:08:00.718154 20271 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:08:00.719143 20271 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:08:00.719199 20271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 23:08:00.719391 20322 tablet_bootstrap.cc:492] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Bootstrap starting.
I20260923 23:08:00.727377 20271 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:33655
I20260923 23:08:00.727476 20429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:33655 every 8 connection(s)
I20260923 23:08:00.727849 20271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:08:00.728027 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 20271
I20260923 23:08:00.728394 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:35375
--local_ip_for_outbound_sockets=127.5.74.2
--tserver_master_addrs=127.5.74.62:34511
--webserver_port=46275
--webserver_interface=127.5.74.2
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:08:00.730715 20322 log.cc:824] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:00.733153 20322 tablet_bootstrap.cc:492] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:08:00.733469 20322 tablet_bootstrap.cc:492] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Bootstrap complete.
I20260923 23:08:00.734439 20322 ts_tablet_manager.cc:1397] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20260923 23:08:00.735513 20322 raft_consensus.cc:348] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.735795 20322 raft_consensus.cc:729] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f583019a264222a3479f05968603c1, State: Initialized, Role: FOLLOWER
I20260923 23:08:00.735944 20322 consensus_queue.cc:260] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:00.736294 20322 ts_tablet_manager.cc:1428] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
W20260923 23:08:00.827852 20432 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:08:00.728539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:46323)
0923 23:08:00.728734 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.728779 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:00.729409 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:00.827466 (+ 98057us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":118,"threads_started":1}
W20260923 23:08:00.828370 20430 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:34511 (0 consecutive failures): Network error: Failed to ping master at 127.5.74.62:34511: Client connection negotiation failed: client connection to 127.5.74.62:34511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 23:08:00.828960 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.828723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:43183)
0923 23:08:00.828826 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.828919 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 23:08:00.829377 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.829270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:56609)
0923 23:08:00.829324 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.829356 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:08:00.829656 20430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:08:00.850430 20433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:00.850695 20433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:00.850728 20433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:00.852336 20433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:00.852403 20433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:00.852437 20433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:08:00.854156 20433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:35375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=46275
--tserver_master_addrs=127.5.74.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:00.854454 20433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:00.854764 20433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:00.860433 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:00.860579 20433 server_base.cc:1034] running on GCE node
W20260923 23:08:00.860466 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:00.860464 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:00.861044 20433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:00.861339 20433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:00.862506 20433 hybrid_clock.cc:648] HybridClock initialized: now 1790204880862483 us; error 37 us; skew 500 ppm
Sep 23 23:08:00 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204880, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:00.865682 20433 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:08:00.866216 20433 webserver.cc:460] Webserver started at http://127.5.74.2:46275/ using document root <none> and password file <none>
I20260923 23:08:00.866463 20433 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:00.866513 20433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:00.868249 20433 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:00.869542 20450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.869940 20433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:08:00.870034 20433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "f0587c36654545f3850069ba68f58d61"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "16da212634d58093162c891d6e85dc7b"
server_key_iv: "56c330984f6f8df01f64a827b0fce42d"
server_key_version: "encryptionkey@0"
I20260923 23:08:00.870471 20433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:00.900424 20433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:00.903071 20433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:00.903232 20433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:00.903540 20433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:00.903877 20433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:00.903936 20433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.903973 20433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:00.904016 20433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:00.911303 20433 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:35375
I20260923 23:08:00.911410 20563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:35375 every 8 connection(s)
I20260923 23:08:00.911832 20433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:08:00.913097 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.912664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:40161)
0923 23:08:00.912854 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.912934 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
W20260923 23:08:00.913605 20564 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:34511 (0 consecutive failures): Network error: Failed to ping master at 127.5.74.62:34511: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
I20260923 23:08:00.914047 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.913937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:42927)
0923 23:08:00.913991 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.914023 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 23:08:00.914423 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:00.914332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:41355)
0923 23:08:00.914382 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:00.914407 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 23:08:00.914556 20564 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:08:00.915889 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 20433
I20260923 23:08:00.916208 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:40903
--local_ip_for_outbound_sockets=127.5.74.3
--tserver_master_addrs=127.5.74.62:34511
--webserver_port=42927
--webserver_interface=127.5.74.3
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:08:01.001817 20567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:01.002027 20567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:01.002053 20567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:01.003397 20567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:01.003475 20567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:01.003506 20567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:08:01.005079 20567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:40903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=42927
--tserver_master_addrs=127.5.74.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:01.005296 20567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:01.005493 20567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:01.011282 20572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:01.011487 20567 server_base.cc:1034] running on GCE node
W20260923 23:08:01.011391 20573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:01.011575 20575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:01.011770 20567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:01.012056 20567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:01.013226 20567 hybrid_clock.cc:648] HybridClock initialized: now 1790204881013190 us; error 48 us; skew 500 ppm
Sep 23 23:08:01 dist-test-slave-j4c5 krb5kdc[19131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204881, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:01.023478 20567 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:08:01.024036 20567 webserver.cc:460] Webserver started at http://127.5.74.3:42927/ using document root <none> and password file <none>
I20260923 23:08:01.024340 20567 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:01.024538 20567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:01.026278 20567 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:08:01.027134 20582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:01.027333 20567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:01.027434 20567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "80de531c397140b5ac3284c31dcb6e53"
format_stamp: "Formatted at 2026-09-23 23:07:59 on dist-test-slave-j4c5"
server_key: "a550e0c60527ccd47a73ae9659da2cb7"
server_key_iv: "f3226b5bdfa2c5df0c6218d25855261f"
server_key_version: "encryptionkey@0"
I20260923 23:08:01.027848 20567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:01.043330 20567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:01.046013 20567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:01.046139 20567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:01.046339 20567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:01.046698 20567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:01.046736 20567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:01.046759 20567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:01.046772 20567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:01.053174 20567 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:40903
I20260923 23:08:01.053221 20695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:40903 every 8 connection(s)
I20260923 23:08:01.053617 20567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:08:01.054761 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:01.054359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:48787)
0923 23:08:01.054535 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:01.054612 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
W20260923 23:08:01.055187 20696 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:34511 (0 consecutive failures): Network error: Failed to ping master at 127.5.74.62:34511: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
I20260923 23:08:01.055505 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:01.055383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:52093)
0923 23:08:01.055447 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:01.055474 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 23:08:01.055768 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:01.055698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:51975)
0923 23:08:01.055728 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:01.055749 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260923 23:08:01.055895 20696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:08:01.062273 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 20567
I20260923 23:08:01.830489 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:01.830114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:59237)
0923 23:08:01.830339 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:01.830414 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
I20260923 23:08:01.915266 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:01.914999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:59353)
0923 23:08:01.915152 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:01.915204 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260923 23:08:02.056624 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:02.056380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:36561)
0923 23:08:02.056517 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:02.056563 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260923 23:08:02.378167 20703 raft_consensus.cc:482] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:08:02.378376 20703 raft_consensus.cc:504] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:02.378852 20703 leader_election.cc:288] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903), f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375)
Sep 23 23:08:02 dist-test-slave-j4c5 krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204880, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.2@KRBTEST.COM
Sep 23 23:08:02 dist-test-slave-j4c5 krb5kdc[19131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204880, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.3@KRBTEST.COM
I20260923 23:08:02.394408 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:02.379653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:58211 (local address 127.5.74.2:35375)
0923 23:08:02.379809 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:02.379817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:02.379845 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:08:02.380333 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:02.380335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:02.380374 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:02.380564 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:02.381290 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:02.382295 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:02.383967 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:02.384768 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:02.384985 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:02.388537 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:02.388544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:02.388544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:02.388562 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:02.391965 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:02.392939 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:02.392944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:02.392945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:02.393093 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:02.393430 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:02.393431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:02.393432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:02.393715 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:02.393810 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:02.394144 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 23:08:02.394276 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260923 23:08:02.394439 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:02.379379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:41769 (local address 127.5.74.3:40903)
0923 23:08:02.379611 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:02.379624 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:02.379667 (+ 43us) server_negotiation.cc:407] Connection header received
0923 23:08:02.380243 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:02.380247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:02.380297 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:02.380426 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:02.381283 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:02.382519 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:02.383951 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:02.384499 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:02.384599 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:02.389103 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:02.389108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:02.389108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:02.389120 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:02.392242 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:02.393000 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:02.393005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:02.393005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:02.393188 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:02.393445 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:02.393446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:02.393446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:02.393715 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:02.393817 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:02.394141 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 23:08:02.394289 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":64,"threads_started":1}
I20260923 23:08:02.394487 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:02.379140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:40903 (local address 127.5.74.1:41769)
0923 23:08:02.379278 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:02.379298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:02.379355 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:02.380527 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:02.380530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:02.380943 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:02.381141 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:02.381149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:02.382621 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:02.382622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:02.383817 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:02.383820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:02.384650 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:02.384652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:02.384867 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:02.385704 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:02.385709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:02.388915 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:02.392372 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:02.392374 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:02.392375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:02.392901 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:02.393256 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:02.393257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:02.393258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:02.393361 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:02.393815 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:02.393816 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:02.393956 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 23:08:02.394068 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 23:08:02.394335 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":189,"thread_start_us":93,"threads_started":1}
I20260923 23:08:02.394657 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:02.379427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:35375 (local address 127.5.74.1:58211)
0923 23:08:02.379535 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:02.379547 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:02.379595 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:02.380541 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:02.380542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:02.380944 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:02.381141 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:02.381149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:02.382470 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:02.382471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:02.383817 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:02.383820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:02.384944 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:02.384945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:02.384996 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:02.385682 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:02.385691 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:02.387903 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:02.392102 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:02.392118 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:02.392122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:02.392792 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:02.393189 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:02.393190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:02.393191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:02.393319 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:02.393801 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:02.393810 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:02.393956 (+ 146us) client_negotiation.cc:769] Sending connection context
0923 23:08:02.394067 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 23:08:02.394582 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":66,"thread_start_us":49,"threads_started":1}
W20260923 23:08:02.395505 20319 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
W20260923 23:08:02.395884 20317 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
I20260923 23:08:02.395951 20317 leader_election.cc:302] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0f583019a264222a3479f05968603c1; no voters: 80de531c397140b5ac3284c31dcb6e53, f0587c36654545f3850069ba68f58d61
I20260923 23:08:02.396209 20703 raft_consensus.cc:2726] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:08:02.831194 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:02.831051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:38537)
0923 23:08:02.831111 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:02.831144 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 23:08:02.916075 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:02.915813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:43197)
0923 23:08:02.915962 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:02.916015 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260923 23:08:03.057462 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:03.057177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:41467)
0923 23:08:03.057361 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:03.057412 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260923 23:08:03.832199 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:03.831920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:36423)
0923 23:08:03.832055 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:03.832102 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260923 23:08:03.916965 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:03.916734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:40039)
0923 23:08:03.916863 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:03.916914 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260923 23:08:03.963169 20712 raft_consensus.cc:482] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:08:03.963284 20712 raft_consensus.cc:504] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:03.963521 20712 leader_election.cc:288] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903), f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375)
W20260923 23:08:03.964102 20317 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
W20260923 23:08:03.964190 20319 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
I20260923 23:08:03.964226 20317 leader_election.cc:302] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0f583019a264222a3479f05968603c1; no voters: 80de531c397140b5ac3284c31dcb6e53, f0587c36654545f3850069ba68f58d61
I20260923 23:08:03.964324 20712 raft_consensus.cc:2726] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:08:04.058192 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:04.057976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:45623)
0923 23:08:04.058100 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:04.058141 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260923 23:08:04.832964 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:04.832725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:40879)
0923 23:08:04.832864 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:04.832912 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260923 23:08:04.917830 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:04.917524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:45227)
0923 23:08:04.917690 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:04.917766 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20260923 23:08:05.059006 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:05.058723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:39493)
0923 23:08:05.058919 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:05.058957 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":108,"threads_started":1}
I20260923 23:08:05.833896 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:05.833638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:34389)
0923 23:08:05.833800 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:05.833846 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20260923 23:08:05.918684 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:05.918421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:53919)
0923 23:08:05.918580 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:05.918630 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260923 23:08:06.059808 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:06.059541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:48801)
0923 23:08:06.059703 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:06.059751 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260923 23:08:06.750475 20720 raft_consensus.cc:482] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:08:06.750576 20720 raft_consensus.cc:504] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:06.750741 20720 leader_election.cc:288] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903), f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375)
W20260923 23:08:06.751179 20317 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
W20260923 23:08:06.751264 20319 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
I20260923 23:08:06.751345 20319 leader_election.cc:302] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0f583019a264222a3479f05968603c1; no voters: 80de531c397140b5ac3284c31dcb6e53, f0587c36654545f3850069ba68f58d61
I20260923 23:08:06.751431 20720 raft_consensus.cc:2726] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:08:06.834645 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:06.834412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:52943)
0923 23:08:06.834558 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:06.834599 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260923 23:08:06.919538 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:06.919283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:45849)
0923 23:08:06.919431 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:06.919476 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260923 23:08:07.060640 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:07.060393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:35303)
0923 23:08:07.060530 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:07.060582 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260923 23:08:07.835609 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:07.835244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:36121)
0923 23:08:07.835503 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:07.835556 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":122,"threads_started":1}
I20260923 23:08:07.920377 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:07.920079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:34957)
0923 23:08:07.920287 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:07.920329 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":115,"threads_started":1}
I20260923 23:08:08.061416 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:08.061179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:44001)
0923 23:08:08.061321 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:08.061365 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260923 23:08:08.836480 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:08.836259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:37115)
0923 23:08:08.836387 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:08.836434 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260923 23:08:08.921195 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:08.920907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:53727)
0923 23:08:08.921100 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:08.921146 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20260923 23:08:09.062254 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:09.061960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:45657)
0923 23:08:09.062156 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:09.062200 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":106,"threads_started":1}
I20260923 23:08:09.749428 20730 raft_consensus.cc:482] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:08:09.749547 20730 raft_consensus.cc:504] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:09.749756 20730 leader_election.cc:288] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903), f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375)
W20260923 23:08:09.750326 20319 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
W20260923 23:08:09.750488 20317 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
I20260923 23:08:09.750543 20317 leader_election.cc:302] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0f583019a264222a3479f05968603c1; no voters: 80de531c397140b5ac3284c31dcb6e53, f0587c36654545f3850069ba68f58d61
I20260923 23:08:09.750643 20730 raft_consensus.cc:2726] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:08:09.837209 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:09.836980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:34315)
0923 23:08:09.837118 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:09.837155 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260923 23:08:09.922158 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:09.921869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:33661)
0923 23:08:09.922025 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:09.922059 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260923 23:08:10.063141 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:10.062845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:49495)
0923 23:08:10.063032 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:10.063077 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260923 23:08:10.838049 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:10.837759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:38097)
0923 23:08:10.837952 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:10.837998 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260923 23:08:10.923159 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:10.922853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:53625)
0923 23:08:10.923059 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:10.923108 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":112,"threads_started":1}
I20260923 23:08:11.064111 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:11.063758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:40683)
0923 23:08:11.063926 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:11.063979 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":66,"threads_started":1}
I20260923 23:08:11.839097 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:11.838750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:44519)
0923 23:08:11.838963 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:11.839013 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
I20260923 23:08:11.924731 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:11.924051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:59151)
0923 23:08:11.924400 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:11.924448 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":76,"threads_started":1}
I20260923 23:08:12.065237 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:12.064819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:34967)
0923 23:08:12.065135 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:12.065179 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260923 23:08:12.839964 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:12.839685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:50925)
0923 23:08:12.839865 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:12.839913 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260923 23:08:12.925664 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:12.925377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:60797)
0923 23:08:12.925551 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:12.925602 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260923 23:08:13.066124 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:13.065846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:47975)
0923 23:08:13.066025 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:13.066073 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260923 23:08:13.840860 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:13.840584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:41507)
0923 23:08:13.840759 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:13.840802 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260923 23:08:13.926834 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:13.926488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:37277)
0923 23:08:13.926700 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:13.926747 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260923 23:08:14.046404 20745 raft_consensus.cc:482] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:08:14.046517 20745 raft_consensus.cc:504] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de531c397140b5ac3284c31dcb6e53" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 40903 } } peers { permanent_uuid: "f0587c36654545f3850069ba68f58d61" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 35375 } } peers { permanent_uuid: "b0f583019a264222a3479f05968603c1" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 33655 } }
I20260923 23:08:14.046710 20745 leader_election.cc:288] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903), f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375)
W20260923 23:08:14.047151 20317 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80de531c397140b5ac3284c31dcb6e53 (127.5.74.3:40903): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
W20260923 23:08:14.047338 20319 leader_election.cc:341] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0587c36654545f3850069ba68f58d61 (127.5.74.2:35375): Not found: Tablet not found: 301d2a2bbac843e597e151cb90e44ede
I20260923 23:08:14.047405 20319 leader_election.cc:302] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0f583019a264222a3479f05968603c1; no voters: 80de531c397140b5ac3284c31dcb6e53, f0587c36654545f3850069ba68f58d61
I20260923 23:08:14.047475 20745 raft_consensus.cc:2726] T 301d2a2bbac843e597e151cb90e44ede P b0f583019a264222a3479f05968603c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:08:14.067149 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:14.066800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:46623)
0923 23:08:14.066961 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:14.067096 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260923 23:08:14.841744 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:14.841468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:33371)
0923 23:08:14.841631 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:14.841693 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260923 23:08:14.927774 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:14.927507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:46345)
0923 23:08:14.927670 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:14.927717 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260923 23:08:15.068203 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:15.067929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:33047)
0923 23:08:15.068077 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:15.068124 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260923 23:08:15.842684 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:15.842370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.1:44747)
0923 23:08:15.842583 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:15.842614 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":153,"threads_started":1}
I20260923 23:08:15.928689 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:15.928484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.2:56623)
0923 23:08:15.928604 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:15.928641 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260923 23:08:16.067241 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 20271
I20260923 23:08:16.069079 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:16.068828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:34511 (local address 127.5.74.3:51479)
0923 23:08:16.068992 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:16.069034 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20260923 23:08:16.072372 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 20433
I20260923 23:08:16.076788 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 20567
2026-09-23T23:08:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18525 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:08:16 dist-test-slave-j4c5 krb5kdc[20755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:08:16 dist-test-slave-j4c5 krb5kdc[20755](info): set up 2 sockets
Sep 23 23:08:16 dist-test-slave-j4c5 krb5kdc[20755](info): commencing operation
krb5kdc: starting...
W20260923 23:08:17.180404 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:08:17 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:08:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:08:17.311476 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37359
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:37359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:08:17.398569 20768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:17.398833 20768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:17.398870 20768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:17.400192 20768 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 23:08:17.400269 20768 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 23:08:17.400290 20768 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 23:08:17.400327 20768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:08:17.400359 20768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:17.400391 20768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:17.400416 20768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:08:17.400446 20768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:08:17.402041 20768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:37359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:17.402450 20768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:17.402746 20768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:17.408723 20774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:17.408802 20768 server_base.cc:1034] running on GCE node
W20260923 23:08:17.408743 20776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:17.408766 20773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:17.409127 20768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:17.409562 20768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:17.410745 20768 hybrid_clock.cc:648] HybridClock initialized: now 1790204897410815 us; error 135 us; skew 500 ppm
Sep 23 23:08:17 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:17.418733 20768 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:08:17.419302 20768 webserver.cc:460] Webserver started at http://127.5.74.62:39777/ using document root <none> and password file <none>
I20260923 23:08:17.419508 20768 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:17.419555 20768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:17.419651 20768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:08:17.420787 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "74952d71ffcf4cfba06584d5d1a42cb7"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "0487867fd1642a8054ad9e8dee4dd08d"
server_key_iv: "58958310972e3862a26dc970fe0bad7e"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.421101 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "74952d71ffcf4cfba06584d5d1a42cb7"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "0487867fd1642a8054ad9e8dee4dd08d"
server_key_iv: "58958310972e3862a26dc970fe0bad7e"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.422480 20768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:08:17.423381 20783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.423626 20768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:17.423715 20768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "74952d71ffcf4cfba06584d5d1a42cb7"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "0487867fd1642a8054ad9e8dee4dd08d"
server_key_iv: "58958310972e3862a26dc970fe0bad7e"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.423777 20768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:17.433997 20768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:17.436636 20768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:17.436787 20768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:17.441171 20768 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:37359
I20260923 23:08:17.441308 20835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:37359 every 8 connection(s)
I20260923 23:08:17.441627 20768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:08:17.442749 20836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:17.445505 20836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Bootstrap starting.
I20260923 23:08:17.446136 20836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:17.446453 20836 log.cc:824] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:17.447126 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 20768
I20260923 23:08:17.447244 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:08:17.447456 5416 external_mini_cluster.cc:1428] Setting key 2eadac55fb4e00aa7e87b4a7c467faa7
I20260923 23:08:17.447472 20836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: No bootstrap required, opened a new log
I20260923 23:08:17.450014 20836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:17.450160 20836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:17.450182 20836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74952d71ffcf4cfba06584d5d1a42cb7, State: Initialized, Role: FOLLOWER
I20260923 23:08:17.450326 20836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:17.450412 20836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:08:17.450459 20836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:08:17.450531 20836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:17.451141 20836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:17.451267 20836 leader_election.cc:302] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74952d71ffcf4cfba06584d5d1a42cb7; no voters:
I20260923 23:08:17.451643 20836 leader_election.cc:288] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:08:17.451712 20841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:08:17.451881 20841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 1 LEADER]: Becoming Leader. State: Replica: 74952d71ffcf4cfba06584d5d1a42cb7, State: Running, Role: LEADER
I20260923 23:08:17.451941 20836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:08:17.452040 20841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:17.452463 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } } }
I20260923 23:08:17.452553 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:08:17.452455 20841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74952d71ffcf4cfba06584d5d1a42cb7. Latest consensus state: current_term: 1 leader_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } } }
I20260923 23:08:17.452626 20841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:08:17.452847 20846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:08:17.453385 20846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 23:08:17 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:08:17.456588 20846 catalog_manager.cc:1347] Generated new cluster ID: bf81dce49d7c4db2956187573b851d3d
I20260923 23:08:17.456647 20846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:08:17.461221 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:17.448105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.5.74.62:37359)
0923 23:08:17.448291 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:17.448299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:17.448366 (+ 67us) server_negotiation.cc:407] Connection header received
0923 23:08:17.448880 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:17.448887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:17.448936 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:17.449055 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:17.449935 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.451496 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.452877 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.453516 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.453632 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:17.456999 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:17.457004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:17.457004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:17.457018 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:17.459302 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:17.459733 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:17.459737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:17.459737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:17.459855 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:17.460065 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:17.460066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:17.460067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:17.460422 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:17.460581 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:17.460670 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 23:08:17.460817 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260923 23:08:17.464967 20846 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:08:17.465667 20846 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:08:17.475190 20846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Generated new TSK 0
I20260923 23:08:17.475407 20846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:08:17.508786 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37359
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:08:17.594414 20862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:17.594676 20862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:17.594718 20862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:17.595942 20862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:17.596028 20862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:17.596077 20862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:08:17.597733 20862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:17.598121 20862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:17.598338 20862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:17.604136 20870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:17.604136 20868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:17.604223 20867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:17.604875 20862 server_base.cc:1034] running on GCE node
I20260923 23:08:17.605121 20862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:17.605394 20862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:17.606585 20862 hybrid_clock.cc:648] HybridClock initialized: now 1790204897606569 us; error 47 us; skew 500 ppm
Sep 23 23:08:17 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:17.609474 20862 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:08:17.609987 20862 webserver.cc:460] Webserver started at http://127.5.74.1:42175/ using document root <none> and password file <none>
I20260923 23:08:17.610296 20862 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:17.610381 20862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:17.610545 20862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:08:17.611706 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "22ab08a72408430fa4497a80180672f2"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "2710fc74a22020fb335d583feb8612d0"
server_key_iv: "f17facbd453885fb6bd56d3e955ae5c7"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.612106 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "22ab08a72408430fa4497a80180672f2"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "2710fc74a22020fb335d583feb8612d0"
server_key_iv: "f17facbd453885fb6bd56d3e955ae5c7"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.613770 20862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:08:17.614686 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.615039 20862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:08:17.615151 20862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "22ab08a72408430fa4497a80180672f2"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "2710fc74a22020fb335d583feb8612d0"
server_key_iv: "f17facbd453885fb6bd56d3e955ae5c7"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.615235 20862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:17.638613 20862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:17.641156 20862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:17.641304 20862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:17.641511 20862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:17.641899 20862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:17.641935 20862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.641995 20862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:17.642023 20862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.648121 20862 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:34767
I20260923 23:08:17.648243 20990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:34767 every 8 connection(s)
I20260923 23:08:17.648605 20862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:08:17.655009 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 20862
I20260923 23:08:17.655164 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:08:17.655390 5416 external_mini_cluster.cc:1428] Setting key 0d3ad65e880a0ad119777215c1ac38fa
Sep 23 23:08:17 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:08:17.659667 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:17.649525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:39983 (local address 127.5.74.62:37359)
0923 23:08:17.649598 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:17.649599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:17.649825 (+ 226us) server_negotiation.cc:407] Connection header received
0923 23:08:17.650331 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:17.650332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:17.650343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:17.650421 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:17.651527 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.652145 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.653371 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.653601 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.653691 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:17.656305 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:17.656307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:17.656307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:17.656316 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:17.658058 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:17.658501 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:17.658502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:17.658502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:17.658569 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:17.658910 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:17.658911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:17.658912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:17.659049 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:17.659123 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:17.659503 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 23:08:17.659594 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:08:17.659966 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:17.649485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:39983)
0923 23:08:17.649660 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:17.649707 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:17.650229 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:17.650479 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:17.650482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:17.651140 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:17.651427 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:17.651432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.652278 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.652279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:17.653266 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:17.653269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.653703 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.653704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:17.653832 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:17.654485 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:17.654496 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:17.656197 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:17.658152 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:17.658158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:17.658160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:17.658413 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:17.658650 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:17.658651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:17.658651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:17.658804 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:17.659133 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:17.659140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:17.659327 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 23:08:17.659447 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 23:08:17.659638 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260923 23:08:17.660768 20991 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37359
I20260923 23:08:17.660874 20991 heartbeater.cc:464] Registering TS with master...
I20260923 23:08:17.661087 20991 heartbeater.cc:511] Master 127.5.74.62:37359 requested a full tablet report, sending...
I20260923 23:08:17.661669 20800 ts_manager.cc:194] Registered new tserver with Master: 22ab08a72408430fa4497a80180672f2 (127.5.74.1:34767)
I20260923 23:08:17.662343 20800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:39983
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:08:17.684708 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37359
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:08:17.769299 20996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:17.769541 20996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:17.769567 20996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:17.770812 20996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:17.770862 20996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:17.770892 20996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:08:17.772589 20996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:17.772811 20996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:17.773011 20996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:17.778259 21004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:17.778429 21002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:17.778258 21001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:17.778777 20996 server_base.cc:1034] running on GCE node
I20260923 23:08:17.779026 20996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:17.779289 20996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:17.780462 20996 hybrid_clock.cc:648] HybridClock initialized: now 1790204897780437 us; error 36 us; skew 500 ppm
Sep 23 23:08:17 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:17.783128 20996 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:08:17.783984 20996 webserver.cc:460] Webserver started at http://127.5.74.2:39503/ using document root <none> and password file <none>
I20260923 23:08:17.784315 20996 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:17.784394 20996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:17.784546 20996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:08:17.785660 20996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "9717ae4977444b91b74514ebb3e335ca"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "7cfe1d2251fbc144aa2803eb8ca28376"
server_key_iv: "61fcfd3b09bf6b7720517275babf31eb"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.786053 20996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "9717ae4977444b91b74514ebb3e335ca"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "7cfe1d2251fbc144aa2803eb8ca28376"
server_key_iv: "61fcfd3b09bf6b7720517275babf31eb"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.787521 20996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:08:17.788539 21011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.788769 20996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:17.788892 20996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "9717ae4977444b91b74514ebb3e335ca"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "7cfe1d2251fbc144aa2803eb8ca28376"
server_key_iv: "61fcfd3b09bf6b7720517275babf31eb"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.788973 20996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:17.814507 20996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:17.817224 20996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:17.817437 20996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:17.817708 20996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:17.818104 20996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:17.818176 20996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.818222 20996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:17.818280 20996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.825536 20996 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:45557
I20260923 23:08:17.825621 21124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:45557 every 8 connection(s)
I20260923 23:08:17.825989 20996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:08:17.830719 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 20996
I20260923 23:08:17.830847 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:08:17.831079 5416 external_mini_cluster.cc:1428] Setting key 56d437087bd1eb6e800229c1a688a95c
Sep 23 23:08:17 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:08:17.836856 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:17.826793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:50455 (local address 127.5.74.62:37359)
0923 23:08:17.826848 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:17.826850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:17.827133 (+ 283us) server_negotiation.cc:407] Connection header received
0923 23:08:17.827653 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:17.827655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:17.827675 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:17.827744 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:17.828874 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.829404 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.830515 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.830946 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.831037 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:17.833406 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:17.833407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:17.833408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:17.833415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:17.835120 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:17.835598 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:17.835600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:17.835600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:17.835676 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:17.836014 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:17.836015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:17.836015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:17.836218 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:17.836296 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:17.836600 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 23:08:17.836713 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:08:17.836894 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:17.826801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:50455)
0923 23:08:17.826988 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:17.827039 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:17.827553 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:17.827849 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:17.827860 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:17.828552 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:17.828765 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:17.828770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.829507 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.829508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:17.830397 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:17.830400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.831057 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.831058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:17.831222 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:17.831829 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:17.831838 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:17.833291 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:17.835216 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:17.835223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:17.835225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:17.835511 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:17.835798 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:17.835798 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:17.835799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:17.835927 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:17.836303 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:17.836309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:17.836447 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 23:08:17.836543 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 23:08:17.836687 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260923 23:08:17.837554 21125 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37359
I20260923 23:08:17.837682 21125 heartbeater.cc:464] Registering TS with master...
I20260923 23:08:17.837911 21125 heartbeater.cc:511] Master 127.5.74.62:37359 requested a full tablet report, sending...
I20260923 23:08:17.838361 20800 ts_manager.cc:194] Registered new tserver with Master: 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
I20260923 23:08:17.838752 20800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:50455
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:08:17.860071 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37359
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:08:17.944715 21130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:17.944941 21130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:17.944968 21130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:17.946260 21130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:17.946326 21130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:17.946357 21130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:08:17.948518 21130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:17.948844 21130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:17.949115 21130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:17.955627 21136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:17.955624 21138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:17.955626 21135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:17.956067 21130 server_base.cc:1034] running on GCE node
I20260923 23:08:17.956331 21130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:17.956643 21130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:17.957847 21130 hybrid_clock.cc:648] HybridClock initialized: now 1790204897957826 us; error 43 us; skew 500 ppm
Sep 23 23:08:17 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:17.960966 21130 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:08:17.961504 21130 webserver.cc:460] Webserver started at http://127.5.74.3:34913/ using document root <none> and password file <none>
I20260923 23:08:17.961766 21130 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:17.961839 21130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:17.961997 21130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:08:17.963115 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "d73a1079bf814192bf19dd63d26c7222"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "214ef45c23e3cdad208213e8147f4bd4"
server_key_iv: "b3e21df466332a86e9248a354671fda7"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.963482 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "d73a1079bf814192bf19dd63d26c7222"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "214ef45c23e3cdad208213e8147f4bd4"
server_key_iv: "b3e21df466332a86e9248a354671fda7"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.965054 21130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:08:17.965952 21145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.966169 21130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:17.966284 21130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "d73a1079bf814192bf19dd63d26c7222"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "214ef45c23e3cdad208213e8147f4bd4"
server_key_iv: "b3e21df466332a86e9248a354671fda7"
server_key_version: "encryptionkey@0"
I20260923 23:08:17.966362 21130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:17.983430 21130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:17.985687 21130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:17.985860 21130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:17.986106 21130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:17.986418 21130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:17.986476 21130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.986521 21130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:17.986544 21130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:17.993069 21130 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:32769
I20260923 23:08:17.993119 21258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:32769 every 8 connection(s)
I20260923 23:08:17.993521 21130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:08:17.996019 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 21130
I20260923 23:08:17.996186 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:08:17.996420 5416 external_mini_cluster.cc:1428] Setting key 0b64de7609c9e7870aa839c23e5561fe
Sep 23 23:08:18 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:08:18.003975 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:17.994485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:60197 (local address 127.5.74.62:37359)
0923 23:08:17.994564 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:17.994567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:17.994817 (+ 250us) server_negotiation.cc:407] Connection header received
0923 23:08:17.995316 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:17.995317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:17.995329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:17.995420 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:17.996529 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.997220 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.998300 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.998509 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.998605 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.000959 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:18.000961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:18.000962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:18.000970 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:18.002444 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:18.002921 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:18.002923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:18.002923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:18.002988 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:18.003257 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:18.003258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:18.003258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:18.003383 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:18.003460 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:18.003763 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 23:08:18.003881 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:08:18.004196 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:17.994437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:60197)
0923 23:08:17.994681 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:17.994722 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:17.995207 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:17.995427 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:17.995431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:17.996067 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:17.996416 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:17.996422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.997317 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.997318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:17.998194 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:17.998196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:17.998612 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:17.998614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:17.998773 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:17.999348 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:17.999357 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:18.000855 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:18.002555 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:18.002564 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:18.002566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:18.002821 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:18.003060 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:18.003061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:18.003061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:18.003184 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:18.003461 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:18.003469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:18.003621 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 23:08:18.003731 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 23:08:18.003905 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
I20260923 23:08:18.004942 21259 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37359
I20260923 23:08:18.005079 21259 heartbeater.cc:464] Registering TS with master...
I20260923 23:08:18.005283 21259 heartbeater.cc:511] Master 127.5.74.62:37359 requested a full tablet report, sending...
I20260923 23:08:18.005703 20800 ts_manager.cc:194] Registered new tserver with Master: d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769)
I20260923 23:08:18.006191 20800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:60197
I20260923 23:08:18.008462 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:08:18.016721 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.009883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.5.74.62:37359)
0923 23:08:18.009979 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:18.009982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:18.010063 (+ 81us) server_negotiation.cc:407] Connection header received
0923 23:08:18.010145 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:18.010146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:18.010157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:18.010241 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:18.010870 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.011359 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.012410 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.012623 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.012722 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.014001 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:18.014003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:18.014003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:18.014011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:18.015401 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:18.015784 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:18.015785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:18.015786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:18.015845 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:18.016072 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:18.016073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:18.016073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:18.016304 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:18.016419 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:18.016522 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 23:08:18.016610 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 23:08:18.018596 20800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:08:18.020351 20800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:08:18.027230 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.022805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:34767 (local address 127.0.0.1:55344)
0923 23:08:18.023289 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:18.023300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:18.023369 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:18.023562 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:18.023565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:18.023567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:18.023636 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.023638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.024734 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.024735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.026007 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.026009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.026826 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.026827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.026918 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.026942 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 23:08:18.027036 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 23:08:18.027097 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260923 23:08:18.027294 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.023033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.5.74.1:34767)
0923 23:08:18.023229 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:18.023235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:18.023392 (+ 157us) server_negotiation.cc:407] Connection header received
0923 23:08:18.023454 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:18.023457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:18.023474 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:18.023542 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:18.023759 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.024631 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.026236 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.026728 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.026814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.026875 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:18.027132 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 23:08:18.027185 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260923 23:08:18.027423 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.023120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.5.74.2:45557)
0923 23:08:18.023306 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:18.023310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:18.023322 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:08:18.023366 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:18.023368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:18.023384 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:18.023478 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:18.023837 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.024980 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.026242 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.026913 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.027006 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.027069 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:18.027275 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 23:08:18.027323 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260923 23:08:18.027647 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.022982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.5.74.3:32769)
0923 23:08:18.023499 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:18.023502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:18.023510 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:08:18.023546 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:18.023548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:18.023564 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:18.023635 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:18.023835 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.025003 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.026139 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.026939 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.027051 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.027451 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:18.027481 (+ 30us) server_negotiation.cc:299] Negotiation successful
0923 23:08:18.027526 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":92,"threads_started":1}
I20260923 23:08:18.027388 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.022808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:32769 (local address 127.0.0.1:54474)
0923 23:08:18.023377 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:18.023387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:18.023428 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:18.023721 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:18.023721 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:18.023723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:18.023753 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.023754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.025106 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.025107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.026065 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.026068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.027152 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.027154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.027211 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.027215 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:08:18.027262 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:08:18.027305 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":3,"thread_start_us":132,"threads_started":1}
I20260923 23:08:18.027230 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.022805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:45557 (local address 127.0.0.1:56942)
0923 23:08:18.023135 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:18.023179 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:18.023249 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:18.023698 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:18.023699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:18.023701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:18.023752 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.023754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.025083 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.025083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.026155 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.026162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.027009 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.027010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.027067 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.027070 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:08:18.027127 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 23:08:18.027159 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":78,"threads_started":1}
I20260923 23:08:18.028467 21193 tablet_service.cc:1467] Processing CreateTablet for tablet 552965cfde2c4781860a3ed8ff8c23df (DEFAULT_TABLE table=test-table [id=5c15331d6c594698906267c46e48f9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:08:18.028465 21059 tablet_service.cc:1467] Processing CreateTablet for tablet 552965cfde2c4781860a3ed8ff8c23df (DEFAULT_TABLE table=test-table [id=5c15331d6c594698906267c46e48f9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:08:18.028697 20925 tablet_service.cc:1467] Processing CreateTablet for tablet 552965cfde2c4781860a3ed8ff8c23df (DEFAULT_TABLE table=test-table [id=5c15331d6c594698906267c46e48f9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:08:18.028832 21059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552965cfde2c4781860a3ed8ff8c23df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:18.028832 21193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552965cfde2c4781860a3ed8ff8c23df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:18.029047 20925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552965cfde2c4781860a3ed8ff8c23df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:18.031579 21276 tablet_bootstrap.cc:492] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Bootstrap starting.
I20260923 23:08:18.032063 21278 tablet_bootstrap.cc:492] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: Bootstrap starting.
I20260923 23:08:18.032644 21276 tablet_bootstrap.cc:654] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:18.032737 21278 tablet_bootstrap.cc:654] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:18.033326 21278 log.cc:824] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:18.033326 21276 log.cc:824] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:18.033672 21277 tablet_bootstrap.cc:492] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: Bootstrap starting.
I20260923 23:08:18.034209 21278 tablet_bootstrap.cc:492] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: No bootstrap required, opened a new log
I20260923 23:08:18.034276 21276 tablet_bootstrap.cc:492] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: No bootstrap required, opened a new log
I20260923 23:08:18.034291 21278 ts_tablet_manager.cc:1397] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:08:18.034328 21277 tablet_bootstrap.cc:654] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:18.034371 21276 ts_tablet_manager.cc:1397] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:08:18.034937 21277 log.cc:824] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:18.035778 21277 tablet_bootstrap.cc:492] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: No bootstrap required, opened a new log
I20260923 23:08:18.035856 21277 ts_tablet_manager.cc:1397] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:08:18.036441 21276 raft_consensus.cc:348] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:18.036616 21276 raft_consensus.cc:374] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:18.036514 21278 raft_consensus.cc:348] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:18.036677 21278 raft_consensus.cc:374] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:18.036684 21276 raft_consensus.cc:729] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ab08a72408430fa4497a80180672f2, State: Initialized, Role: FOLLOWER
I20260923 23:08:18.036780 21278 raft_consensus.cc:729] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73a1079bf814192bf19dd63d26c7222, State: Initialized, Role: FOLLOWER
I20260923 23:08:18.036829 21276 consensus_queue.cc:260] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:18.036932 21278 consensus_queue.cc:260] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:18.037176 20991 heartbeater.cc:503] Master 127.5.74.62:37359 was elected leader, sending a full tablet report...
I20260923 23:08:18.037240 21278 ts_tablet_manager.cc:1428] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:08:18.037367 21259 heartbeater.cc:503] Master 127.5.74.62:37359 was elected leader, sending a full tablet report...
I20260923 23:08:18.037176 21276 ts_tablet_manager.cc:1428] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:08:18.038089 21277 raft_consensus.cc:348] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:18.038285 21277 raft_consensus.cc:374] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:18.038324 21277 raft_consensus.cc:729] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9717ae4977444b91b74514ebb3e335ca, State: Initialized, Role: FOLLOWER
I20260923 23:08:18.038447 21277 consensus_queue.cc:260] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:18.038753 21277 ts_tablet_manager.cc:1428] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:08:18.038910 21125 heartbeater.cc:503] Master 127.5.74.62:37359 was elected leader, sending a full tablet report...
W20260923 23:08:18.076637 21126 tablet.cc:2369] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:08:18.149518 20992 tablet.cc:2369] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:08:18.237861 21282 raft_consensus.cc:482] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:08:18.238004 21282 raft_consensus.cc:504] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:18.238328 21282 leader_election.cc:288] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769), 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
I20260923 23:08:18.241871 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.238608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:32769 (local address 127.5.74.1:38381)
0923 23:08:18.238771 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:18.238784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:18.238868 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:18.239134 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:18.239136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:18.239139 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:18.239223 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.239226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.240011 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.240012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.240988 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.240991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.241609 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.241610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.241688 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.241693 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:08:18.241754 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:08:18.241790 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260923 23:08:18.242026 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.238775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:38381 (local address 127.5.74.3:32769)
0923 23:08:18.238826 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:18.238828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:18.238847 (+ 19us) server_negotiation.cc:407] Connection header received
0923 23:08:18.239030 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:18.239032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:18.239048 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:18.239115 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:18.239311 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.239876 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.241086 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.241521 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.241582 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.241591 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:18.241879 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 23:08:18.241930 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:08:18.242290 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.238728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:55809 (local address 127.5.74.2:45557)
0923 23:08:18.238779 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:18.238782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:18.238873 (+ 91us) server_negotiation.cc:407] Connection header received
0923 23:08:18.239061 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:18.239062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:18.239075 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:18.239154 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:18.239402 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.240230 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.241188 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.241588 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.241662 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.241674 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:18.242170 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 23:08:18.242224 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:08:18.242287 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.238608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:45557 (local address 127.5.74.1:55809)
0923 23:08:18.238768 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:18.238781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:18.238865 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:18.239183 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:18.239184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:18.239186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:18.239309 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.239311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.240326 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.240327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.241098 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:18.241101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.241884 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.241885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:18.241957 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.241961 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:08:18.242157 (+ 196us) client_negotiation.cc:240] Negotiation successful
0923 23:08:18.242203 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260923 23:08:18.242362 21213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552965cfde2c4781860a3ed8ff8c23df" candidate_uuid: "22ab08a72408430fa4497a80180672f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73a1079bf814192bf19dd63d26c7222" is_pre_election: true
I20260923 23:08:18.242563 21213 raft_consensus.cc:2454] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ab08a72408430fa4497a80180672f2 in term 0.
I20260923 23:08:18.242631 21079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552965cfde2c4781860a3ed8ff8c23df" candidate_uuid: "22ab08a72408430fa4497a80180672f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9717ae4977444b91b74514ebb3e335ca" is_pre_election: true
I20260923 23:08:18.242775 21079 raft_consensus.cc:2454] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ab08a72408430fa4497a80180672f2 in term 0.
I20260923 23:08:18.242871 20879 leader_election.cc:302] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ab08a72408430fa4497a80180672f2, d73a1079bf814192bf19dd63d26c7222; no voters:
I20260923 23:08:18.243072 21282 raft_consensus.cc:2781] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:08:18.243112 21282 raft_consensus.cc:482] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:08:18.243129 21282 raft_consensus.cc:3037] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:18.243856 21282 raft_consensus.cc:504] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:18.244025 21282 leader_election.cc:288] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 1 election: Requested vote from peers d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769), 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
W20260923 23:08:18.244129 21260 tablet.cc:2369] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:08:18.244365 21213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552965cfde2c4781860a3ed8ff8c23df" candidate_uuid: "22ab08a72408430fa4497a80180672f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73a1079bf814192bf19dd63d26c7222"
I20260923 23:08:18.244491 21213 raft_consensus.cc:3037] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:18.244516 21079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552965cfde2c4781860a3ed8ff8c23df" candidate_uuid: "22ab08a72408430fa4497a80180672f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9717ae4977444b91b74514ebb3e335ca"
I20260923 23:08:18.244602 21079 raft_consensus.cc:3037] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:18.245503 21079 raft_consensus.cc:2454] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ab08a72408430fa4497a80180672f2 in term 1.
I20260923 23:08:18.245503 21213 raft_consensus.cc:2454] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ab08a72408430fa4497a80180672f2 in term 1.
I20260923 23:08:18.245733 20881 leader_election.cc:302] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ab08a72408430fa4497a80180672f2, 9717ae4977444b91b74514ebb3e335ca; no voters:
I20260923 23:08:18.245834 21282 raft_consensus.cc:2781] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:08:18.246037 21282 raft_consensus.cc:686] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 1 LEADER]: Becoming Leader. State: Replica: 22ab08a72408430fa4497a80180672f2, State: Running, Role: LEADER
I20260923 23:08:18.246124 21282 consensus_queue.cc:237] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:18.247077 20798 catalog_manager.cc:5495] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ab08a72408430fa4497a80180672f2 (127.5.74.1). New cstate: current_term: 1 leader_uuid: "22ab08a72408430fa4497a80180672f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } health_report { overall_health: UNKNOWN } } }
I20260923 23:08:18.259783 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.257180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.5.74.1:34767)
0923 23:08:18.257256 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:18.257258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:18.257276 (+ 18us) server_negotiation.cc:407] Connection header received
0923 23:08:18.257418 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:18.257419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:18.257444 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:18.257526 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:08:18.257685 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.258182 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.258955 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:18.259150 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:18.259231 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:18.259413 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:08:18.259540 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:08:18.259638 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:18.259658 (+ 20us) server_negotiation.cc:299] Negotiation successful
0923 23:08:18.259682 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 23:08:18.261780 21079 raft_consensus.cc:1261] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [term 1 FOLLOWER]: Refusing update from remote peer 22ab08a72408430fa4497a80180672f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:08:18.261780 21213 raft_consensus.cc:1261] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [term 1 FOLLOWER]: Refusing update from remote peer 22ab08a72408430fa4497a80180672f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:08:18.262183 21287 consensus_queue.cc:1035] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:08:18.262327 21282 consensus_queue.cc:1035] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:08:18.265516 21290 mvcc.cc:205] Tried to move back new op lower bound from 7332679263277281280 to 7332679263216750592. Current Snapshot: MvccSnapshot[applied={T|T < 7332679263277281280}]
I20260923 23:08:18.267143 21291 mvcc.cc:205] Tried to move back new op lower bound from 7332679263277281280 to 7332679263216750592. Current Snapshot: MvccSnapshot[applied={T|T < 7332679263277281280}]
I20260923 23:08:18.268522 20800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:08:18.268630 20800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:08:18.269510 20800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Sending DeleteTablet for 3 replicas of tablet 552965cfde2c4781860a3ed8ff8c23df
I20260923 23:08:18.269930 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 20862
I20260923 23:08:18.269914 20925 tablet_service.cc:1514] Processing DeleteTablet for tablet 552965cfde2c4781860a3ed8ff8c23df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:08:18 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:55344
I20260923 23:08:18.270018 21193 tablet_service.cc:1514] Processing DeleteTablet for tablet 552965cfde2c4781860a3ed8ff8c23df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:08:18 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:54474
I20260923 23:08:18.270285 21059 tablet_service.cc:1514] Processing DeleteTablet for tablet 552965cfde2c4781860a3ed8ff8c23df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:08:18 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260923 23:08:18.270722 21301 tablet_replica.cc:330] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: stopping tablet replica
I20260923 23:08:18.270838 21301 raft_consensus.cc:2229] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:08:18.270951 21301 raft_consensus.cc:2258] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:08:18.272068 21301 ts_tablet_manager.cc:1905] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:08:18.273962 21301 ts_tablet_manager.cc:1918] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:08:18.274073 21301 log.cc:1196] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/552965cfde2c4781860a3ed8ff8c23df
I20260923 23:08:18.274336 21301 ts_tablet_manager.cc:1939] T 552965cfde2c4781860a3ed8ff8c23df P 9717ae4977444b91b74514ebb3e335ca: Deleting consensus metadata
I20260923 23:08:18.274886 20787 catalog_manager.cc:4841] TS 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557): tablet 552965cfde2c4781860a3ed8ff8c23df (table test-table [id=5c15331d6c594698906267c46e48f9bc]) successfully deleted
W20260923 23:08:18.275554 20784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:34767 (error 108)
I20260923 23:08:18.275804 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 20996
I20260923 23:08:18.275835 21300 tablet_replica.cc:330] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: stopping tablet replica
I20260923 23:08:18.275949 21300 raft_consensus.cc:2229] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:08:18.276072 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:18.275936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:34767 (local address 127.0.0.1:55354)
0923 23:08:18.276001 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:18.276035 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 23:08:18.276124 21300 raft_consensus.cc:2258] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 23:08:18.276252 20784 catalog_manager.cc:4584] TS 22ab08a72408430fa4497a80180672f2 (127.5.74.1:34767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 552965cfde2c4781860a3ed8ff8c23df: Network error: Client connection negotiation failed: client connection to 127.5.74.1:34767: connect: Connection refused (error 111)
I20260923 23:08:18.276746 21300 ts_tablet_manager.cc:1905] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:08:18.278520 21300 ts_tablet_manager.cc:1918] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:08:18.278646 21300 log.cc:1196] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/552965cfde2c4781860a3ed8ff8c23df
I20260923 23:08:18.278963 21300 ts_tablet_manager.cc:1939] T 552965cfde2c4781860a3ed8ff8c23df P d73a1079bf814192bf19dd63d26c7222: Deleting consensus metadata
I20260923 23:08:18.279520 20785 catalog_manager.cc:4841] TS d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769): tablet 552965cfde2c4781860a3ed8ff8c23df (table test-table [id=5c15331d6c594698906267c46e48f9bc]) successfully deleted
I20260923 23:08:18.282032 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 21130
I20260923 23:08:18.287067 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 20768
I20260923 23:08:38.294953 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37359
--webserver_interface=127.5.74.62
--webserver_port=39777
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:37359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:08:38.387188 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:38.387440 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:38.387468 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:38.388872 21303 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 23:08:38.388927 21303 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 23:08:38.388942 21303 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 23:08:38.388954 21303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:08:38.388967 21303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:38.388981 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:38.389024 21303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:08:38.389057 21303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:08:38.390995 21303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:37359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=39777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:38.391319 21303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:38.391672 21303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:38.401858 21312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:38.401858 21309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:38.401858 21310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:38.402171 21303 server_base.cc:1034] running on GCE node
I20260923 23:08:38.402457 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:38.402817 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:38.403988 21303 hybrid_clock.cc:648] HybridClock initialized: now 1790204918403950 us; error 50 us; skew 500 ppm
Sep 23 23:08:38 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:38.406936 21303 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:08:38.407382 21303 webserver.cc:460] Webserver started at http://127.5.74.62:39777/ using document root <none> and password file <none>
I20260923 23:08:38.407653 21303 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:38.407743 21303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:38.410015 21303 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:38.411044 21319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:38.411360 21303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:38.411505 21303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "74952d71ffcf4cfba06584d5d1a42cb7"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "0487867fd1642a8054ad9e8dee4dd08d"
server_key_iv: "58958310972e3862a26dc970fe0bad7e"
server_key_version: "encryptionkey@0"
I20260923 23:08:38.412112 21303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:38.421221 21303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:38.423650 21303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:38.423841 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:38.428861 21303 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:37359
I20260923 23:08:38.428913 21371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:37359 every 8 connection(s)
I20260923 23:08:38.429352 21303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:08:38.431037 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 21303
I20260923 23:08:38.431252 21372 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:08:38.431411 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:34767
--local_ip_for_outbound_sockets=127.5.74.1
--tserver_master_addrs=127.5.74.62:37359
--webserver_port=42175
--webserver_interface=127.5.74.1
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:08:38.432088 21372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Bootstrap starting.
I20260923 23:08:38.441532 21372 log.cc:824] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:38.443756 21372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:08:38.444145 21372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Bootstrap complete.
I20260923 23:08:38.447069 21372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:38.447492 21372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74952d71ffcf4cfba06584d5d1a42cb7, State: Initialized, Role: FOLLOWER
I20260923 23:08:38.447713 21372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:38.447817 21372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:08:38.447881 21372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:08:38.447958 21372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:08:38.449148 21372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:38.449355 21372 leader_election.cc:302] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74952d71ffcf4cfba06584d5d1a42cb7; no voters:
I20260923 23:08:38.449616 21372 leader_election.cc:288] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 23:08:38.449671 21376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:08:38.450057 21376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 2 LEADER]: Becoming Leader. State: Replica: 74952d71ffcf4cfba06584d5d1a42cb7, State: Running, Role: LEADER
I20260923 23:08:38.450181 21376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:38.450032 21372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:08:38.450515 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74952d71ffcf4cfba06584d5d1a42cb7. Latest consensus state: current_term: 2 leader_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } } }
I20260923 23:08:38.450603 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:08:38.450806 21377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } } }
I20260923 23:08:38.450863 21377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:08:38.450964 21380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:08:38.451747 21380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:08:38.451843 21380 catalog_manager.cc:1259] Loaded cluster ID: bf81dce49d7c4db2956187573b851d3d
I20260923 23:08:38.451879 21380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:08:38.453117 21380 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:08:38.453328 21380 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Loaded TSK: 0
I20260923 23:08:38.466840 21380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Generated new TSK 1
I20260923 23:08:38.466988 21380 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:08:38.529292 21374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:38.529522 21374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:38.529549 21374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:38.530843 21374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:38.530902 21374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:38.530933 21374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:08:38.533273 21374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:34767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=42175
--tserver_master_addrs=127.5.74.62:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:38.533581 21374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:38.534109 21374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:38.541081 21399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:38.541062 21400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:38.541064 21402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:38.541462 21374 server_base.cc:1034] running on GCE node
I20260923 23:08:38.541678 21374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:38.541999 21374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:38.543210 21374 hybrid_clock.cc:648] HybridClock initialized: now 1790204918543181 us; error 43 us; skew 500 ppm
Sep 23 23:08:38 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:38.546281 21374 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:08:38.546960 21374 webserver.cc:460] Webserver started at http://127.5.74.1:42175/ using document root <none> and password file <none>
I20260923 23:08:38.547212 21374 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:38.547295 21374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:38.548949 21374 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:08:38.549856 21409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:38.550060 21374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:38.550170 21374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "22ab08a72408430fa4497a80180672f2"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "2710fc74a22020fb335d583feb8612d0"
server_key_iv: "f17facbd453885fb6bd56d3e955ae5c7"
server_key_version: "encryptionkey@0"
I20260923 23:08:38.550698 21374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:38.601485 21374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:38.604487 21374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:38.604658 21374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:38.604960 21374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:38.605463 21416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:08:38.607193 21374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:08:38.607257 21374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:08:38.607286 21374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:08:38.608227 21374 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:08:38.608300 21374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260923 23:08:38.608305 21416 tablet_bootstrap.cc:492] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Bootstrap starting.
I20260923 23:08:38.617981 21374 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:34767
I20260923 23:08:38.618062 21523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:34767 every 8 connection(s)
I20260923 23:08:38.618564 21374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:08:38.618752 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 21374
I20260923 23:08:38.619102 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:45557
--local_ip_for_outbound_sockets=127.5.74.2
--tserver_master_addrs=127.5.74.62:37359
--webserver_port=39503
--webserver_interface=127.5.74.2
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:08:38.620430 21416 log.cc:824] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:38.623006 21416 tablet_bootstrap.cc:492] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:08:38.623381 21416 tablet_bootstrap.cc:492] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Bootstrap complete.
I20260923 23:08:38.624447 21416 ts_tablet_manager.cc:1397] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.015s
I20260923 23:08:38.626163 21416 raft_consensus.cc:348] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:38.626724 21416 raft_consensus.cc:729] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ab08a72408430fa4497a80180672f2, State: Initialized, Role: FOLLOWER
I20260923 23:08:38.626916 21416 consensus_queue.cc:260] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } } peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } }
I20260923 23:08:38.627300 21416 ts_tablet_manager.cc:1428] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
Sep 23 23:08:38 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:08:38.632809 21527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.619715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:37049 (local address 127.5.74.62:37359)
0923 23:08:38.620322 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:38.620333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:38.620383 (+ 50us) server_negotiation.cc:407] Connection header received
0923 23:08:38.621196 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:38.621202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:38.621260 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:38.621408 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:38.622104 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.623015 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.624225 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.624929 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.625060 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.628452 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:38.628456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:38.628457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:38.628470 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:38.630445 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.631124 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.631127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.631127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.631234 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.631572 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.631573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.631574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.631869 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:38.631942 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:38.632533 (+ 591us) server_negotiation.cc:299] Negotiation successful
0923 23:08:38.632663 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":93,"threads_started":1}
I20260923 23:08:38.632819 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.619592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:37049)
0923 23:08:38.620041 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:38.620090 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:38.620915 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:38.621437 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:38.621441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:38.621772 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:38.621965 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.621971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.623157 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.623159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.624086 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.624090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.625080 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.625082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.625240 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.625989 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:38.626001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:38.628276 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:38.630561 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:38.630572 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:38.630574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:38.630983 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:38.631316 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:38.631317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:38.631317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:38.631478 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:38.631955 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:38.631964 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:38.632181 (+ 217us) client_negotiation.cc:769] Sending connection context
0923 23:08:38.632314 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 23:08:38.632541 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":204,"threads_started":1}
I20260923 23:08:38.633778 21524 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37359
I20260923 23:08:38.633905 21524 heartbeater.cc:464] Registering TS with master...
I20260923 23:08:38.634166 21524 heartbeater.cc:511] Master 127.5.74.62:37359 requested a full tablet report, sending...
I20260923 23:08:38.635053 21336 ts_manager.cc:194] Registered new tserver with Master: 22ab08a72408430fa4497a80180672f2 (127.5.74.1:34767)
I20260923 23:08:38.636407 21336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:37049
I20260923 23:08:38.637450 21524 heartbeater.cc:503] Master 127.5.74.62:37359 was elected leader, sending a full tablet report...
Sep 23 23:08:38 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.1@KRBTEST.COM
I20260923 23:08:38.645605 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.635757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:34767 (local address 127.0.0.1:39454)
0923 23:08:38.636037 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:38.636075 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:38.636225 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:38.636661 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:38.636685 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:38.637070 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:38.637137 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.637141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.638431 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.638432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.639058 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.639061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.639666 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.639670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.639740 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.640323 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:38.640328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:38.642101 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:38.644285 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:38.644293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:38.644295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:38.644533 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:38.644796 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:38.644797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:38.644797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:38.644897 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:38.645253 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:38.645260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:38.645315 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 23:08:38.645393 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 23:08:38.645515 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":123,"threads_started":1}
I20260923 23:08:38.645658 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.635804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.5.74.1:34767)
0923 23:08:38.635996 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:38.636018 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:38.636193 (+ 175us) server_negotiation.cc:407] Connection header received
0923 23:08:38.636422 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:38.636428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:38.636450 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:38.636581 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:38.637225 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.638340 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.639135 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.639587 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.639655 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.642207 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:38.642218 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:38.642219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:38.642227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:38.644187 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.644620 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.644624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.644624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.644697 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.644976 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.644977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.644977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.645176 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:38.645264 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:38.645445 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 23:08:38.645550 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
I20260923 23:08:38.646235 21455 tablet_service.cc:1514] Processing DeleteTablet for tablet 552965cfde2c4781860a3ed8ff8c23df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:08:18 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:39454
I20260923 23:08:38.646895 21533 tablet_replica.cc:330] stopping tablet replica
I20260923 23:08:38.647019 21533 raft_consensus.cc:2229] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:08:38.647119 21533 raft_consensus.cc:2258] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:08:38.647461 21533 ts_tablet_manager.cc:1905] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:08:38.650207 21533 ts_tablet_manager.cc:1918] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:08:38.650333 21533 log.cc:1196] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal/wals/552965cfde2c4781860a3ed8ff8c23df
I20260923 23:08:38.650665 21533 ts_tablet_manager.cc:1939] T 552965cfde2c4781860a3ed8ff8c23df P 22ab08a72408430fa4497a80180672f2: Deleting consensus metadata
I20260923 23:08:38.651417 21320 catalog_manager.cc:4841] TS 22ab08a72408430fa4497a80180672f2 (127.5.74.1:34767): tablet 552965cfde2c4781860a3ed8ff8c23df (table test-table [id=5c15331d6c594698906267c46e48f9bc]) successfully deleted
W20260923 23:08:38.715521 21528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:38.715773 21528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:38.715817 21528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:38.717231 21528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:38.717322 21528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:38.717372 21528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:08:38.719002 21528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:45557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=39503
--tserver_master_addrs=127.5.74.62:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:38.719309 21528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:38.719564 21528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:38.724963 21538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:38.724967 21541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:38.724963 21539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:38.725317 21528 server_base.cc:1034] running on GCE node
I20260923 23:08:38.725662 21528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:38.725937 21528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:38.727125 21528 hybrid_clock.cc:648] HybridClock initialized: now 1790204918727118 us; error 40 us; skew 500 ppm
Sep 23 23:08:38 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:38.730144 21528 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:08:38.730718 21528 webserver.cc:460] Webserver started at http://127.5.74.2:39503/ using document root <none> and password file <none>
I20260923 23:08:38.730979 21528 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:38.731060 21528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:38.733078 21528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:08:38.733904 21548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:38.734133 21528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:38.734256 21528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "9717ae4977444b91b74514ebb3e335ca"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "7cfe1d2251fbc144aa2803eb8ca28376"
server_key_iv: "61fcfd3b09bf6b7720517275babf31eb"
server_key_version: "encryptionkey@0"
I20260923 23:08:38.734696 21528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:38.758258 21528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:38.760814 21528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:38.760993 21528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:38.761264 21528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:38.761595 21528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:38.761648 21528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:38.761683 21528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:38.761711 21528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:38.767516 21528 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:45557
I20260923 23:08:38.767606 21661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:45557 every 8 connection(s)
I20260923 23:08:38.767968 21528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:08:38.768471 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 21528
I20260923 23:08:38.768748 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:32769
--local_ip_for_outbound_sockets=127.5.74.3
--tserver_master_addrs=127.5.74.62:37359
--webserver_port=34913
--webserver_interface=127.5.74.3
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:08:38 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:08:38.780828 21527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.768938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:33827 (local address 127.5.74.62:37359)
0923 23:08:38.768993 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:38.768995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:38.769287 (+ 292us) server_negotiation.cc:407] Connection header received
0923 23:08:38.769810 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:38.769812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:38.769827 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:38.769920 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:38.771520 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.772371 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.773698 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.774228 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.774329 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.777207 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:38.777209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:38.777210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:38.777219 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:38.779143 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.779656 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.779658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.779659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.779731 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.779985 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.779987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.779988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.780196 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:38.780279 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:38.780576 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 23:08:38.780689 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:08:38.780846 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.768958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:33827)
0923 23:08:38.769127 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:38.769173 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:38.769708 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:38.770017 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:38.770022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:38.771017 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:38.771374 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.771382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.772495 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.772496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.773572 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.773575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.774374 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.774376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.774516 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.775231 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:38.775240 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:38.777097 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:38.779247 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:38.779255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:38.779257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:38.779570 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:38.779808 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:38.779808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:38.779809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:38.779916 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:38.780270 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:38.780276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:38.780431 (+ 155us) client_negotiation.cc:769] Sending connection context
0923 23:08:38.780525 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 23:08:38.780663 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260923 23:08:38.781703 21662 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37359
I20260923 23:08:38.781832 21662 heartbeater.cc:464] Registering TS with master...
I20260923 23:08:38.782070 21662 heartbeater.cc:511] Master 127.5.74.62:37359 requested a full tablet report, sending...
I20260923 23:08:38.782589 21336 ts_manager.cc:194] Registered new tserver with Master: 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
I20260923 23:08:38.783138 21336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:33827
W20260923 23:08:38.864823 21665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:38.865065 21665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:38.865093 21665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:38.866412 21665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:38.866477 21665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:38.866509 21665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:08:38.868178 21665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:32769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=34913
--tserver_master_addrs=127.5.74.62:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:38.868486 21665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:38.868899 21665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:38.874805 21670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:38.874805 21673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:38.874848 21671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:38.875290 21665 server_base.cc:1034] running on GCE node
I20260923 23:08:38.875528 21665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:38.875823 21665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:38.876994 21665 hybrid_clock.cc:648] HybridClock initialized: now 1790204918876946 us; error 64 us; skew 500 ppm
Sep 23 23:08:38 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:38.880266 21665 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:08:38.880836 21665 webserver.cc:460] Webserver started at http://127.5.74.3:34913/ using document root <none> and password file <none>
I20260923 23:08:38.881222 21665 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:38.881340 21665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:38.882887 21665 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:08:38.883658 21680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:38.883980 21665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:38.884114 21665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "d73a1079bf814192bf19dd63d26c7222"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "214ef45c23e3cdad208213e8147f4bd4"
server_key_iv: "b3e21df466332a86e9248a354671fda7"
server_key_version: "encryptionkey@0"
I20260923 23:08:38.884564 21665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:38.903664 21665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:38.906785 21665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:38.907011 21665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:38.907274 21665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:38.907615 21665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:38.907671 21665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:38.907706 21665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:38.907734 21665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:38.914971 21665 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:32769
I20260923 23:08:38.915037 21793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:32769 every 8 connection(s)
I20260923 23:08:38.915513 21665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:08:38.915764 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 21665
Sep 23 23:08:38 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:08:38.923942 21527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.916404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.5.74.62:37359)
0923 23:08:38.916484 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:38.916486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:38.916514 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:08:38.916629 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:38.916630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:38.916640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:38.916692 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:38.917352 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.917852 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.918624 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.918835 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.918922 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.920511 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:38.920513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:38.920513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:38.920521 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:38.922455 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.922922 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.922924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.922924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.922990 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.923221 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.923222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.923222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.923362 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:38.923685 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:38.923784 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 23:08:38.923857 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:08:38.927074 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.916462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:54951 (local address 127.5.74.62:37359)
0923 23:08:38.916624 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:38.916627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:38.916815 (+ 188us) server_negotiation.cc:407] Connection header received
0923 23:08:38.917366 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:38.917367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:38.917380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:38.917460 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:38.918464 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.918955 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.920189 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.920398 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.920474 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.923089 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:38.923091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:38.923091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:38.923098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:38.925434 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.925958 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.925959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.925959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.926031 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:38.926313 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:38.926314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:38.926315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:38.926478 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:38.926548 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:38.926900 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 23:08:38.926994 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":49,"threads_started":1}
I20260923 23:08:38.927141 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.916407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:54951)
0923 23:08:38.916659 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:38.916699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:38.917252 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:38.917536 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:38.917541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:38.918115 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:08:38.918359 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.918365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.919057 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.919059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.920070 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.920073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.920496 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.920498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.920620 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.921173 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:08:38.921182 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:08:38.922976 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:08:38.925554 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:38.925562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:38.925564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:38.925838 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:38.926105 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:08:38.926105 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:08:38.926106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:08:38.926222 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:08:38.926562 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:08:38.926568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:08:38.926700 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 23:08:38.926801 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 23:08:38.926942 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260923 23:08:38.927953 21794 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37359
I20260923 23:08:38.928073 21794 heartbeater.cc:464] Registering TS with master...
I20260923 23:08:38.928313 21794 heartbeater.cc:511] Master 127.5.74.62:37359 requested a full tablet report, sending...
I20260923 23:08:38.928934 21336 ts_manager.cc:194] Registered new tserver with Master: d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769)
I20260923 23:08:38.929327 21336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:54951
I20260923 23:08:38.936769 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:08:38.940583 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.937182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.5.74.62:37359)
0923 23:08:38.937277 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:38.937283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:38.937391 (+ 108us) server_negotiation.cc:407] Connection header received
0923 23:08:38.937470 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:38.937471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:38.937527 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:38.937614 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:08:38.937818 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.938450 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.939392 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.939731 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.939801 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.940088 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:08:38.940269 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:08:38.940352 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:38.940428 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 23:08:38.940461 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 23:08:38.940951 21336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:08:38.942668 21336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:08:38.945515 21455 tablet_service.cc:1467] Processing CreateTablet for tablet 44bc3b44e5d84c4ba9114797c3b9f113 (DEFAULT_TABLE table=test-table [id=99a5221d64f54536ac965ac06c9ac4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:08:38.945775 21455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bc3b44e5d84c4ba9114797c3b9f113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:38.947381 21416 tablet_bootstrap.cc:492] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Bootstrap starting.
I20260923 23:08:38.947783 21416 tablet_bootstrap.cc:654] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:38.948452 21416 tablet_bootstrap.cc:492] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: No bootstrap required, opened a new log
I20260923 23:08:38.948513 21416 ts_tablet_manager.cc:1397] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:08:38.948906 21416 raft_consensus.cc:348] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:38.949018 21416 raft_consensus.cc:374] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:38.949084 21416 raft_consensus.cc:729] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ab08a72408430fa4497a80180672f2, State: Initialized, Role: FOLLOWER
I20260923 23:08:38.949141 21416 consensus_queue.cc:260] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:38.949261 21416 ts_tablet_manager.cc:1428] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 23:08:38.950119 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.945240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:45557 (local address 127.0.0.1:57842)
0923 23:08:38.945579 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:38.945591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:38.945666 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:38.946229 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:38.946230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:38.946233 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:38.946280 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.946281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.947583 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.947584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.948727 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.948731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.949834 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.949835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.949903 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.949925 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 23:08:38.949969 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 23:08:38.950015 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":90,"threads_started":1}
I20260923 23:08:38.950202 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.945096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:32769 (local address 127.0.0.1:51762)
0923 23:08:38.945208 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:38.945232 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:38.945269 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:38.945812 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:38.945814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:38.945816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:38.945970 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.945974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.947518 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.947520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.948909 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:38.948913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.949978 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.949979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:38.950037 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.950041 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:08:38.950094 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:08:38.950137 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":59}
I20260923 23:08:38.950285 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.945239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.5.74.3:32769)
0923 23:08:38.945475 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:38.945482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:38.945588 (+ 106us) server_negotiation.cc:407] Connection header received
0923 23:08:38.945653 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:38.945655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:38.945674 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:38.946076 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:38.946203 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.947369 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.949014 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.949819 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.949881 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.949945 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:38.950118 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 23:08:38.950177 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":164,"threads_started":1}
I20260923 23:08:38.950507 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:38.945661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.5.74.2:45557)
0923 23:08:38.945971 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:38.945976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:38.945989 (+ 13us) server_negotiation.cc:407] Connection header received
0923 23:08:38.946052 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:38.946055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:38.946070 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:38.946148 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:38.946355 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.947501 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.948825 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:38.949698 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:38.950198 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:38.950282 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:38.950311 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 23:08:38.950420 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260923 23:08:38.951186 21728 tablet_service.cc:1467] Processing CreateTablet for tablet 44bc3b44e5d84c4ba9114797c3b9f113 (DEFAULT_TABLE table=test-table [id=99a5221d64f54536ac965ac06c9ac4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:08:38.951321 21596 tablet_service.cc:1467] Processing CreateTablet for tablet 44bc3b44e5d84c4ba9114797c3b9f113 (DEFAULT_TABLE table=test-table [id=99a5221d64f54536ac965ac06c9ac4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:08:38.951576 21728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bc3b44e5d84c4ba9114797c3b9f113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:38.951756 21596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bc3b44e5d84c4ba9114797c3b9f113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:08:38.954905 21805 tablet_bootstrap.cc:492] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: Bootstrap starting.
I20260923 23:08:38.954810 21804 tablet_bootstrap.cc:492] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: Bootstrap starting.
I20260923 23:08:38.955785 21804 tablet_bootstrap.cc:654] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:38.955785 21805 tablet_bootstrap.cc:654] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: Neither blocks nor log segments found. Creating new log.
I20260923 23:08:38.956629 21804 log.cc:824] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:38.956629 21805 log.cc:824] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:38.957491 21805 tablet_bootstrap.cc:492] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: No bootstrap required, opened a new log
I20260923 23:08:38.957495 21804 tablet_bootstrap.cc:492] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: No bootstrap required, opened a new log
I20260923 23:08:38.957587 21805 ts_tablet_manager.cc:1397] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:08:38.957608 21804 ts_tablet_manager.cc:1397] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:08:38.959626 21805 raft_consensus.cc:348] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:38.959699 21804 raft_consensus.cc:348] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:38.959851 21804 raft_consensus.cc:374] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:38.959856 21805 raft_consensus.cc:374] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:08:38.959875 21804 raft_consensus.cc:729] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73a1079bf814192bf19dd63d26c7222, State: Initialized, Role: FOLLOWER
I20260923 23:08:38.959913 21805 raft_consensus.cc:729] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9717ae4977444b91b74514ebb3e335ca, State: Initialized, Role: FOLLOWER
I20260923 23:08:38.960047 21804 consensus_queue.cc:260] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:38.960055 21805 consensus_queue.cc:260] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:38.960434 21794 heartbeater.cc:503] Master 127.5.74.62:37359 was elected leader, sending a full tablet report...
I20260923 23:08:38.960430 21662 heartbeater.cc:503] Master 127.5.74.62:37359 was elected leader, sending a full tablet report...
I20260923 23:08:38.960441 21805 ts_tablet_manager.cc:1428] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:08:38.960724 21804 ts_tablet_manager.cc:1428] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
W20260923 23:08:39.018649 21663 tablet.cc:2369] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:08:39.119539 21525 tablet.cc:2369] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:08:39.166848 21795 tablet.cc:2369] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:08:39.228483 21530 raft_consensus.cc:482] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:08:39.228749 21530 raft_consensus.cc:504] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:39.229326 21530 leader_election.cc:288] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769), 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
I20260923 23:08:39.232576 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:39.229676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:41663 (local address 127.5.74.2:45557)
0923 23:08:39.229725 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:39.229728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:39.229873 (+ 145us) server_negotiation.cc:407] Connection header received
0923 23:08:39.229938 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:39.229940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:39.229951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:39.230031 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:39.230174 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.230654 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.231678 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.232187 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.232285 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:39.232297 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:39.232465 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 23:08:39.232503 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:08:39.232584 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:39.229598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:45557 (local address 127.5.74.1:41663)
0923 23:08:39.229812 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:39.229821 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:39.229874 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:39.230033 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:39.230034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:39.230036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:39.230101 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:39.230103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.230743 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.230745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:39.231575 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:39.231578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.232277 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.232278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:39.232357 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:39.232379 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 23:08:39.232445 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 23:08:39.232495 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":34,"thread_start_us":136,"threads_started":1}
I20260923 23:08:39.232681 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:39.229616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:43521 (local address 127.5.74.3:32769)
0923 23:08:39.229759 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:39.229762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:39.229775 (+ 13us) server_negotiation.cc:407] Connection header received
0923 23:08:39.229909 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:39.229910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:39.229922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:39.230035 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:08:39.230212 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.230691 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.231785 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.232315 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.232396 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:39.232407 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:39.232574 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 23:08:39.232617 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260923 23:08:39.232686 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:39.229520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:32769 (local address 127.5.74.1:43521)
0923 23:08:39.229676 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:39.229689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:39.229786 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:39.230046 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:08:39.230047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:08:39.230051 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:08:39.230132 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:39.230134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.230777 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.230779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:39.231662 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:08:39.231673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.232410 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.232411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:08:39.232488 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:39.232492 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:08:39.232555 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 23:08:39.232600 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260923 23:08:39.233222 21748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bc3b44e5d84c4ba9114797c3b9f113" candidate_uuid: "22ab08a72408430fa4497a80180672f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73a1079bf814192bf19dd63d26c7222" is_pre_election: true
I20260923 23:08:39.233323 21616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bc3b44e5d84c4ba9114797c3b9f113" candidate_uuid: "22ab08a72408430fa4497a80180672f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9717ae4977444b91b74514ebb3e335ca" is_pre_election: true
I20260923 23:08:39.233424 21748 raft_consensus.cc:2454] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ab08a72408430fa4497a80180672f2 in term 0.
I20260923 23:08:39.233588 21616 raft_consensus.cc:2454] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ab08a72408430fa4497a80180672f2 in term 0.
I20260923 23:08:39.233692 21411 leader_election.cc:302] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ab08a72408430fa4497a80180672f2, d73a1079bf814192bf19dd63d26c7222; no voters:
I20260923 23:08:39.233827 21530 raft_consensus.cc:2781] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:08:39.233897 21530 raft_consensus.cc:482] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:08:39.233929 21530 raft_consensus.cc:3037] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:39.234607 21530 raft_consensus.cc:504] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:39.234763 21530 leader_election.cc:288] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 1 election: Requested vote from peers d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769), 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
I20260923 23:08:39.234947 21616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bc3b44e5d84c4ba9114797c3b9f113" candidate_uuid: "22ab08a72408430fa4497a80180672f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9717ae4977444b91b74514ebb3e335ca"
I20260923 23:08:39.234946 21748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bc3b44e5d84c4ba9114797c3b9f113" candidate_uuid: "22ab08a72408430fa4497a80180672f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73a1079bf814192bf19dd63d26c7222"
I20260923 23:08:39.235028 21616 raft_consensus.cc:3037] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:39.235029 21748 raft_consensus.cc:3037] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:08:39.235944 21748 raft_consensus.cc:2454] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ab08a72408430fa4497a80180672f2 in term 1.
I20260923 23:08:39.235936 21616 raft_consensus.cc:2454] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ab08a72408430fa4497a80180672f2 in term 1.
I20260923 23:08:39.236209 21413 leader_election.cc:302] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ab08a72408430fa4497a80180672f2, 9717ae4977444b91b74514ebb3e335ca; no voters:
I20260923 23:08:39.236305 21530 raft_consensus.cc:2781] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:08:39.236542 21530 raft_consensus.cc:686] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 LEADER]: Becoming Leader. State: Replica: 22ab08a72408430fa4497a80180672f2, State: Running, Role: LEADER
I20260923 23:08:39.236624 21530 consensus_queue.cc:237] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:39.237638 21335 catalog_manager.cc:5495] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ab08a72408430fa4497a80180672f2 (127.5.74.1). New cstate: current_term: 1 leader_uuid: "22ab08a72408430fa4497a80180672f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } health_report { overall_health: HEALTHY } } }
I20260923 23:08:39.309031 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:39.306370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.5.74.1:34767)
0923 23:08:39.306550 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:39.306552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:39.306559 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:08:39.306607 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:39.306607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:39.306627 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:39.306719 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:08:39.306871 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.307367 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.308144 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.308432 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.308517 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:39.308714 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:08:39.308768 (+ 54us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 23:08:39.308940 (+ 172us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56530: token expired
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
W20260923 23:08:39.309118 21814 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56530: token expired
I20260923 23:08:39.316928 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:39.309578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.5.74.62:37359)
0923 23:08:39.309654 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:39.309655 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:39.309663 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:08:39.309687 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:39.309688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:39.309696 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:39.309749 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:08:39.310223 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.310840 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.311611 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.311796 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.311860 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:39.313480 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:08:39.313482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:08:39.313482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:08:39.313489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:08:39.315159 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:39.315619 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:39.315621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:39.315621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:39.315723 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:08:39.316048 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:08:39.316049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:08:39.316049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:08:39.316304 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:08:39.316399 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:39.316707 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 23:08:39.316830 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:08:39.320387 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:39.318089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.5.74.1:34767)
0923 23:08:39.318156 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:08:39.318158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:08:39.318165 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:08:39.318199 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:08:39.318200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:08:39.318209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:08:39.318335 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:08:39.318381 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.318860 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.319563 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:08:39.319766 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:08:39.319851 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:08:39.320014 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:08:39.320121 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:08:39.320201 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 23:08:39.320266 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 23:08:39.320296 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:08:39.322019 21616 raft_consensus.cc:1261] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [term 1 FOLLOWER]: Refusing update from remote peer 22ab08a72408430fa4497a80180672f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:08:39.322021 21748 raft_consensus.cc:1261] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [term 1 FOLLOWER]: Refusing update from remote peer 22ab08a72408430fa4497a80180672f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:08:39.322492 21812 consensus_queue.cc:1035] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:08:39.322621 21530 consensus_queue.cc:1035] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:08:39.325424 21815 mvcc.cc:205] Tried to move back new op lower bound from 7332679349540315136 to 7332679349194125312. Current Snapshot: MvccSnapshot[applied={T|T < 7332679349540315136}]
I20260923 23:08:39.325913 21817 mvcc.cc:205] Tried to move back new op lower bound from 7332679349540315136 to 7332679349194125312. Current Snapshot: MvccSnapshot[applied={T|T < 7332679349540315136}]
I20260923 23:08:39.329286 21336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:08:39.329401 21336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:08:39.330258 21336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Sending DeleteTablet for 3 replicas of tablet 44bc3b44e5d84c4ba9114797c3b9f113
I20260923 23:08:39.330736 21728 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bc3b44e5d84c4ba9114797c3b9f113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:08:39 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:51762
I20260923 23:08:39.330771 21596 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bc3b44e5d84c4ba9114797c3b9f113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:08:39 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:57842
I20260923 23:08:39.331020 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 21374
I20260923 23:08:39.331158 21825 tablet_replica.cc:330] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: stopping tablet replica
I20260923 23:08:39.331137 21455 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bc3b44e5d84c4ba9114797c3b9f113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:08:39 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:39454
I20260923 23:08:39.331341 21825 raft_consensus.cc:2229] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:08:39.331588 21825 raft_consensus.cc:2258] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:08:39.331686 21824 tablet_replica.cc:330] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: stopping tablet replica
I20260923 23:08:39.331804 21824 raft_consensus.cc:2229] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:08:39.331907 21824 raft_consensus.cc:2258] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:08:39.332398 21825 ts_tablet_manager.cc:1905] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:08:39.332405 21824 ts_tablet_manager.cc:1905] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:08:39.334527 21824 ts_tablet_manager.cc:1918] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:08:39.334527 21825 ts_tablet_manager.cc:1918] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:08:39.334623 21824 log.cc:1196] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/44bc3b44e5d84c4ba9114797c3b9f113
I20260923 23:08:39.334623 21825 log.cc:1196] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/44bc3b44e5d84c4ba9114797c3b9f113
I20260923 23:08:39.334920 21825 ts_tablet_manager.cc:1939] T 44bc3b44e5d84c4ba9114797c3b9f113 P 9717ae4977444b91b74514ebb3e335ca: Deleting consensus metadata
I20260923 23:08:39.334960 21824 ts_tablet_manager.cc:1939] T 44bc3b44e5d84c4ba9114797c3b9f113 P d73a1079bf814192bf19dd63d26c7222: Deleting consensus metadata
I20260923 23:08:39.335422 21321 catalog_manager.cc:4841] TS d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769): tablet 44bc3b44e5d84c4ba9114797c3b9f113 (table test-table [id=99a5221d64f54536ac965ac06c9ac4cf]) successfully deleted
I20260923 23:08:39.335479 21323 catalog_manager.cc:4841] TS 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557): tablet 44bc3b44e5d84c4ba9114797c3b9f113 (table test-table [id=99a5221d64f54536ac965ac06c9ac4cf]) successfully deleted
W20260923 23:08:39.337024 21320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:34767 (error 108)
I20260923 23:08:39.337602 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 21528
I20260923 23:08:39.337617 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:39.337443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:34767 (local address 127.0.0.1:56542)
0923 23:08:39.337509 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:39.337577 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 23:08:39.339115 21320 catalog_manager.cc:4584] TS 22ab08a72408430fa4497a80180672f2 (127.5.74.1:34767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44bc3b44e5d84c4ba9114797c3b9f113: Network error: Client connection negotiation failed: client connection to 127.5.74.1:34767: connect: Connection refused (error 111)
I20260923 23:08:39.342856 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 21665
I20260923 23:08:39.347534 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 21303
I20260923 23:08:59.354012 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37359
--webserver_interface=127.5.74.62
--webserver_port=39777
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:37359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:08:59.440032 21827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:59.440299 21827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:59.440325 21827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:59.441560 21827 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 23:08:59.441601 21827 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 23:08:59.441614 21827 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 23:08:59.441625 21827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:08:59.441639 21827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:59.441653 21827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:59.441665 21827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:08:59.441677 21827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:08:59.443482 21827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:37359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=39777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:59.443737 21827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:59.444044 21827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:59.450592 21833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:59.450665 21827 server_base.cc:1034] running on GCE node
W20260923 23:08:59.450615 21835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:59.450614 21832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:59.451313 21827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:59.451690 21827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:59.452872 21827 hybrid_clock.cc:648] HybridClock initialized: now 1790204939452854 us; error 38 us; skew 500 ppm
Sep 23 23:08:59 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204939, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:59.463019 21827 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:08:59.463544 21827 webserver.cc:460] Webserver started at http://127.5.74.62:39777/ using document root <none> and password file <none>
I20260923 23:08:59.463804 21827 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:59.463874 21827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:59.465472 21827 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:59.466410 21842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:59.466655 21827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:59.466784 21827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "74952d71ffcf4cfba06584d5d1a42cb7"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "0487867fd1642a8054ad9e8dee4dd08d"
server_key_iv: "58958310972e3862a26dc970fe0bad7e"
server_key_version: "encryptionkey@0"
I20260923 23:08:59.467276 21827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:59.503803 21827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:59.506425 21827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:59.506626 21827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:59.511945 21827 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:37359
I20260923 23:08:59.511979 21894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:37359 every 8 connection(s)
I20260923 23:08:59.512449 21827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:08:59.514122 21895 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:08:59.514901 21895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Bootstrap starting.
I20260923 23:08:59.520376 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 21827
I20260923 23:08:59.520682 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:34767
--local_ip_for_outbound_sockets=127.5.74.1
--tserver_master_addrs=127.5.74.62:37359
--webserver_port=42175
--webserver_interface=127.5.74.1
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:08:59.524842 21895 log.cc:824] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:59.527673 21895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:08:59.528087 21895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Bootstrap complete.
I20260923 23:08:59.530759 21895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:59.531039 21895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74952d71ffcf4cfba06584d5d1a42cb7, State: Initialized, Role: FOLLOWER
I20260923 23:08:59.531229 21895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:59.531349 21895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:08:59.531416 21895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:08:59.531502 21895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 2 FOLLOWER]: Advancing to term 3
I20260923 23:08:59.532900 21895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:59.533042 21895 leader_election.cc:302] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74952d71ffcf4cfba06584d5d1a42cb7; no voters:
I20260923 23:08:59.533289 21895 leader_election.cc:288] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 23:08:59.533324 21899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 23:08:59.533492 21899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [term 3 LEADER]: Becoming Leader. State: Replica: 74952d71ffcf4cfba06584d5d1a42cb7, State: Running, Role: LEADER
I20260923 23:08:59.533625 21895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:08:59.533681 21899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } }
I20260923 23:08:59.534126 21900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } } }
I20260923 23:08:59.534237 21900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:08:59.534181 21901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74952d71ffcf4cfba06584d5d1a42cb7. Latest consensus state: current_term: 3 leader_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74952d71ffcf4cfba06584d5d1a42cb7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37359 } } }
I20260923 23:08:59.534286 21901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:08:59.534525 21906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:08:59.535564 21906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:08:59.535704 21906 catalog_manager.cc:1259] Loaded cluster ID: bf81dce49d7c4db2956187573b851d3d
I20260923 23:08:59.535744 21906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:08:59.536937 21906 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:08:59.537151 21906 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Loaded TSK: 0
I20260923 23:08:59.537215 21906 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74952d71ffcf4cfba06584d5d1a42cb7: Loaded TSK: 1
F20260923 23:08:59.537470 21906 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff0e448b9ad google::LogMessage::Fail() at ??:0
@ 0x7ff0e448e88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff0e448b4de google::LogMessage::Flush() at ??:0
@ 0x7ff0e448ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff0e6bf8104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff0e6bf5d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff0ec3e51cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff0ec3e548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff0ec3e5594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff0ec3abc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff0ec3b5ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff0e63590d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff0e634e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff0e71b26db start_thread at ??:0
@ 0x7ff0e2f8961f clone at ??:0
W20260923 23:08:59.640751 21897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:59.640978 21897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:59.641006 21897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:59.642306 21897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:59.642362 21897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:59.642393 21897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:08:59.644011 21897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:34767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=42175
--tserver_master_addrs=127.5.74.62:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:59.644328 21897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:59.644593 21897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:59.651044 21936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:59.651157 21937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:59.651125 21897 server_base.cc:1034] running on GCE node
W20260923 23:08:59.651054 21939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:59.651654 21897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:59.651968 21897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:59.653167 21897 hybrid_clock.cc:648] HybridClock initialized: now 1790204939653140 us; error 47 us; skew 500 ppm
Sep 23 23:08:59 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204939, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:59.656371 21897 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:08:59.656886 21897 webserver.cc:460] Webserver started at http://127.5.74.1:42175/ using document root <none> and password file <none>
I20260923 23:08:59.657140 21897 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:59.657209 21897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:59.658742 21897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:59.659570 21946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:59.659765 21897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:59.659866 21897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "22ab08a72408430fa4497a80180672f2"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "2710fc74a22020fb335d583feb8612d0"
server_key_iv: "f17facbd453885fb6bd56d3e955ae5c7"
server_key_version: "encryptionkey@0"
I20260923 23:08:59.660322 21897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:59.690246 21897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:59.692948 21897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:59.693097 21897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:59.693383 21897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:59.693900 21953 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:08:59.695528 21897 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:08:59.695588 21897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:08:59.695617 21897 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:08:59.696409 21897 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:08:59.696462 21897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260923 23:08:59.696509 21953 tablet_bootstrap.cc:492] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Bootstrap starting.
I20260923 23:08:59.704571 21897 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:34767
I20260923 23:08:59.704726 22060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:34767 every 8 connection(s)
I20260923 23:08:59.705157 21897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:08:59.707434 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 21897
I20260923 23:08:59.707809 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:45557
--local_ip_for_outbound_sockets=127.5.74.2
--tserver_master_addrs=127.5.74.62:37359
--webserver_port=39503
--webserver_interface=127.5.74.2
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:08:59.709651 21953 log.cc:824] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Log is configured to *not* fsync() on all Append() calls
I20260923 23:08:59.712286 21953 tablet_bootstrap.cc:492] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:08:59.712618 21953 tablet_bootstrap.cc:492] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Bootstrap complete.
I20260923 23:08:59.713595 21953 ts_tablet_manager.cc:1397] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20260923 23:08:59.714699 21953 raft_consensus.cc:348] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:59.714962 21953 raft_consensus.cc:729] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ab08a72408430fa4497a80180672f2, State: Initialized, Role: FOLLOWER
I20260923 23:08:59.715138 21953 consensus_queue.cc:260] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:08:59.715437 21953 ts_tablet_manager.cc:1428] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260923 23:08:59.782770 22063 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:08:59.706019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:41525)
0923 23:08:59.706205 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:59.706257 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 23:08:59.706948 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:08:59.782549 (+ 75601us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
W20260923 23:08:59.783043 22061 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:37359 (0 consecutive failures): Network error: Failed to ping master at 127.5.74.62:37359: Client connection negotiation failed: client connection to 127.5.74.62:37359: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 23:08:59.783392 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:59.783269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:41827)
0923 23:08:59.783321 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:59.783371 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 23:08:59.783689 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:59.783598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:45669)
0923 23:08:59.783640 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:59.783666 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:08:59.783809 22061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:08:59.815445 22064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:59.815677 22064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:59.815706 22064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:59.817199 22064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:59.817260 22064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:59.817293 22064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:08:59.818933 22064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:45557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=39503
--tserver_master_addrs=127.5.74.62:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:59.819204 22064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:59.819450 22064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:08:59.825222 22064 server_base.cc:1034] running on GCE node
W20260923 23:08:59.825187 22072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:59.825187 22074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:59.825222 22071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:59.825798 22064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:59.826058 22064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:59.827224 22064 hybrid_clock.cc:648] HybridClock initialized: now 1790204939827209 us; error 30 us; skew 500 ppm
Sep 23 23:08:59 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204939, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:59.829937 22064 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:08:59.830416 22064 webserver.cc:460] Webserver started at http://127.5.74.2:39503/ using document root <none> and password file <none>
I20260923 23:08:59.830646 22064 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:59.830694 22064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:59.832324 22064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:08:59.833102 22081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:59.833410 22064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:59.833496 22064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "9717ae4977444b91b74514ebb3e335ca"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "7cfe1d2251fbc144aa2803eb8ca28376"
server_key_iv: "61fcfd3b09bf6b7720517275babf31eb"
server_key_version: "encryptionkey@0"
I20260923 23:08:59.833923 22064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:59.848591 22064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:59.851212 22064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:59.851420 22064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:59.851667 22064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:59.852025 22064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:59.852062 22064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:59.852085 22064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:59.852098 22064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:59.858441 22064 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:45557
I20260923 23:08:59.858578 22194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:45557 every 8 connection(s)
I20260923 23:08:59.858880 22064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:08:59.860145 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:59.859761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:60729)
0923 23:08:59.859912 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:59.859988 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
W20260923 23:08:59.860620 22195 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:37359 (0 consecutive failures): Network error: Failed to ping master at 127.5.74.62:37359: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
I20260923 23:08:59.860973 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:59.860836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:51755)
0923 23:08:59.860912 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:59.860946 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 23:08:59.861347 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:59.861240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:37477)
0923 23:08:59.861304 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:59.861330 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 23:08:59.861536 22195 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:08:59.865085 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 22064
I20260923 23:08:59.865522 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:32769
--local_ip_for_outbound_sockets=127.5.74.3
--tserver_master_addrs=127.5.74.62:37359
--webserver_port=34913
--webserver_interface=127.5.74.3
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:08:59.951195 22198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:08:59.951493 22198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:08:59.951547 22198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:08:59.952983 22198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:08:59.953076 22198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:08:59.953132 22198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:08:59.954986 22198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:32769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=34913
--tserver_master_addrs=127.5.74.62:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:08:59.955312 22198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:08:59.955601 22198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:08:59.961627 22204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:59.961638 22203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:08:59.961642 22206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:08:59.962056 22198 server_base.cc:1034] running on GCE node
I20260923 23:08:59.962328 22198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:08:59.962615 22198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:08:59.963819 22198 hybrid_clock.cc:648] HybridClock initialized: now 1790204939963787 us; error 32 us; skew 500 ppm
Sep 23 23:08:59 dist-test-slave-j4c5 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204939, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:08:59.966776 22198 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:08:59.967255 22198 webserver.cc:460] Webserver started at http://127.5.74.3:34913/ using document root <none> and password file <none>
I20260923 23:08:59.967499 22198 fs_manager.cc:359] Metadata directory not provided
I20260923 23:08:59.967567 22198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:08:59.969059 22198 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:59.969838 22213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:59.970055 22198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:08:59.970160 22198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "d73a1079bf814192bf19dd63d26c7222"
format_stamp: "Formatted at 2026-09-23 23:08:17 on dist-test-slave-j4c5"
server_key: "214ef45c23e3cdad208213e8147f4bd4"
server_key_iv: "b3e21df466332a86e9248a354671fda7"
server_key_version: "encryptionkey@0"
I20260923 23:08:59.970569 22198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:08:59.983974 22198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:08:59.986719 22198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:08:59.986941 22198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:08:59.987221 22198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:08:59.987576 22198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:08:59.987633 22198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:59.987679 22198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:08:59.987726 22198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:08:59.994349 22198 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:32769
I20260923 23:08:59.994400 22326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:32769 every 8 connection(s)
I20260923 23:08:59.994758 22198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:08:59.995868 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:59.995474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:54111)
0923 23:08:59.995627 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:59.995704 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
W20260923 23:08:59.996372 22327 heartbeater.cc:644] Failed to heartbeat to 127.5.74.62:37359 (0 consecutive failures): Network error: Failed to ping master at 127.5.74.62:37359: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
I20260923 23:08:59.996721 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:59.996582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:45621)
0923 23:08:59.996659 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:59.996692 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 23:08:59.997056 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:08:59.996966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:41305)
0923 23:08:59.997012 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:08:59.997037 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 23:08:59.997187 22327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:09:00.001458 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 22198
I20260923 23:09:00.784526 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:00.784272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:49075)
0923 23:09:00.784430 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:00.784465 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260923 23:09:00.862274 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:00.861965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:54111)
0923 23:09:00.862131 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:00.862182 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260923 23:09:00.997846 22333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:00.997568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:55271)
0923 23:09:00.997749 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:00.997787 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260923 23:09:01.119450 22334 raft_consensus.cc:482] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:01.119585 22334 raft_consensus.cc:504] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:09:01.120071 22334 leader_election.cc:288] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769), 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
Sep 23 23:09:01 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204939, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.2@KRBTEST.COM
Sep 23 23:09:01 dist-test-slave-j4c5 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204939, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.3@KRBTEST.COM
I20260923 23:09:01.135390 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:01.120553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:35255 (local address 127.5.74.2:45557)
0923 23:09:01.120723 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:01.120733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:01.120777 (+ 44us) server_negotiation.cc:407] Connection header received
0923 23:09:01.121488 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:01.121492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:01.121546 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:01.121690 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:01.122586 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:01.124146 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:01.125853 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:01.126491 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:01.126590 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:01.130668 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:01.130673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:01.130673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:01.130701 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:01.133118 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:01.133973 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:01.133978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:01.133978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:01.134097 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:01.134447 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:01.134448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:01.134449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:01.134718 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:01.134807 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:01.135108 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 23:09:01.135256 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260923 23:09:01.135386 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:01.120318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:45557 (local address 127.5.74.1:35255)
0923 23:09:01.120686 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:01.120698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:01.120770 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:01.121693 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:01.121694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:01.122166 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:01.122464 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:01.122473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:01.124324 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:01.124326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:01.125736 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:01.125741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:01.126678 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:01.126678 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:01.126723 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:01.127412 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:01.127423 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:01.130476 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:01.133279 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:01.133288 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:01.133291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:01.133734 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:01.134187 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:01.134188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:01.134189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:01.134324 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:01.134796 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:01.134802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:01.134944 (+ 142us) client_negotiation.cc:769] Sending connection context
0923 23:09:01.135053 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 23:09:01.135269 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":296,"thread_start_us":54,"threads_started":1}
I20260923 23:09:01.135407 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:01.120517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:49971 (local address 127.5.74.3:32769)
0923 23:09:01.120721 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:01.120733 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:01.120773 (+ 40us) server_negotiation.cc:407] Connection header received
0923 23:09:01.121282 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:01.121284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:01.121333 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:01.121517 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:01.122627 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:01.124289 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:01.125866 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:01.126435 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:01.126552 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:01.130646 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:01.130653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:01.130653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:01.130682 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:01.133158 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:01.134051 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:01.134055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:01.134055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:01.134165 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:01.134428 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:01.134429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:01.134429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:01.134716 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:01.134807 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:01.135105 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 23:09:01.135286 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260923 23:09:01.135571 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:01.120318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:32769 (local address 127.5.74.1:49971)
0923 23:09:01.120411 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:01.120425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:01.120505 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:01.121563 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:01.121566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:01.122147 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:01.122464 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:01.122472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:01.124378 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:01.124379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:01.125736 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:01.125740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:01.126542 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:01.126543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:01.126670 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:01.127476 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:01.127481 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:01.130467 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:01.133277 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:01.133288 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:01.133291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:01.133927 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:01.134242 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:01.134243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:01.134243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:01.134343 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:01.134794 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:01.134801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:01.134945 (+ 144us) client_negotiation.cc:769] Sending connection context
0923 23:09:01.135054 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 23:09:01.135350 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":185}
W20260923 23:09:01.136916 21948 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
W20260923 23:09:01.136997 21950 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
I20260923 23:09:01.137075 21950 leader_election.cc:302] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ab08a72408430fa4497a80180672f2; no voters: 9717ae4977444b91b74514ebb3e335ca, d73a1079bf814192bf19dd63d26c7222
I20260923 23:09:01.137179 22334 raft_consensus.cc:2726] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:09:01.785450 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:01.785132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:53153)
0923 23:09:01.785321 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:01.785402 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260923 23:09:01.863131 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:01.862894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:49437)
0923 23:09:01.863038 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:01.863084 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260923 23:09:01.998818 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:01.998524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:41039)
0923 23:09:01.998687 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:01.998728 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260923 23:09:02.786379 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:02.786110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:58437)
0923 23:09:02.786284 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:02.786331 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260923 23:09:02.805577 22342 raft_consensus.cc:482] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:02.805694 22342 raft_consensus.cc:504] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:09:02.805908 22342 leader_election.cc:288] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769), 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
W20260923 23:09:02.806371 21948 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
W20260923 23:09:02.806459 21950 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
I20260923 23:09:02.806494 21950 leader_election.cc:302] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ab08a72408430fa4497a80180672f2; no voters: 9717ae4977444b91b74514ebb3e335ca, d73a1079bf814192bf19dd63d26c7222
I20260923 23:09:02.806581 22342 raft_consensus.cc:2726] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:09:02.864419 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:02.863756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:58945)
0923 23:09:02.864117 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:02.864320 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":270,"threads_started":1}
I20260923 23:09:02.999717 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:02.999446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:39947)
0923 23:09:02.999620 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:02.999667 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
I20260923 23:09:03.787300 22345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:03.787022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:38157)
0923 23:09:03.787200 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:03.787251 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20260923 23:09:03.865222 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:03.864973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:58637)
0923 23:09:03.865125 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:03.865172 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260923 23:09:04.000594 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:04.000326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:59553)
0923 23:09:04.000489 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:04.000536 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260923 23:09:04.343393 22348 raft_consensus.cc:482] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:04.343536 22348 raft_consensus.cc:504] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:09:04.343750 22348 leader_election.cc:288] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769), 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
W20260923 23:09:04.344257 21948 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
W20260923 23:09:04.344331 21950 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
I20260923 23:09:04.344398 21950 leader_election.cc:302] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ab08a72408430fa4497a80180672f2; no voters: 9717ae4977444b91b74514ebb3e335ca, d73a1079bf814192bf19dd63d26c7222
I20260923 23:09:04.344489 22348 raft_consensus.cc:2726] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:09:04.788281 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:04.787954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:52553)
0923 23:09:04.788137 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:04.788229 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260923 23:09:04.866056 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:04.865823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:58363)
0923 23:09:04.865961 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:04.866008 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260923 23:09:05.001397 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:05.001117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:59237)
0923 23:09:05.001292 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:05.001346 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
I20260923 23:09:05.789273 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:05.788954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:38649)
0923 23:09:05.789167 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:05.789222 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":120,"threads_started":1}
I20260923 23:09:05.866954 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:05.866646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:55229)
0923 23:09:05.866857 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:05.866902 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
I20260923 23:09:06.002333 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:06.002043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:48295)
0923 23:09:06.002238 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:06.002285 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":110,"threads_started":1}
I20260923 23:09:06.790234 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:06.789964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:53891)
0923 23:09:06.790139 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:06.790187 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260923 23:09:06.868052 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:06.867623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:39297)
0923 23:09:06.867783 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:06.868002 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260923 23:09:07.003376 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:07.003082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:39675)
0923 23:09:07.003273 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:07.003321 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
I20260923 23:09:07.791177 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:07.790864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:38481)
0923 23:09:07.791073 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:07.791123 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260923 23:09:07.869091 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:07.868834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:60599)
0923 23:09:07.868993 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:07.869040 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260923 23:09:08.004345 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:08.004018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:59335)
0923 23:09:08.004238 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:08.004288 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260923 23:09:08.397397 22361 raft_consensus.cc:482] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:08.397531 22361 raft_consensus.cc:504] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:09:08.397734 22361 leader_election.cc:288] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769), 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
W20260923 23:09:08.398214 21950 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
W20260923 23:09:08.398293 21948 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
I20260923 23:09:08.398391 21948 leader_election.cc:302] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ab08a72408430fa4497a80180672f2; no voters: 9717ae4977444b91b74514ebb3e335ca, d73a1079bf814192bf19dd63d26c7222
I20260923 23:09:08.398524 22361 raft_consensus.cc:2726] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:09:08.792119 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:08.791810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:58573)
0923 23:09:08.792020 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:08.792068 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
I20260923 23:09:08.870206 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:08.869949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:42691)
0923 23:09:08.870091 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:08.870141 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260923 23:09:09.005337 22364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:09.004979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:42817)
0923 23:09:09.005167 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:09.005234 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260923 23:09:09.793035 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:09.792751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:60265)
0923 23:09:09.792928 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:09.792983 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260923 23:09:09.871153 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:09.870882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:46929)
0923 23:09:09.871031 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:09.871084 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260923 23:09:10.006245 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:10.006010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:51041)
0923 23:09:10.006147 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:10.006194 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260923 23:09:10.630201 22368 raft_consensus.cc:482] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:10.630316 22368 raft_consensus.cc:504] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73a1079bf814192bf19dd63d26c7222" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 32769 } } peers { permanent_uuid: "9717ae4977444b91b74514ebb3e335ca" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 45557 } } peers { permanent_uuid: "22ab08a72408430fa4497a80180672f2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 34767 } }
I20260923 23:09:10.630615 22368 leader_election.cc:288] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769), 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557)
W20260923 23:09:10.631165 21950 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9717ae4977444b91b74514ebb3e335ca (127.5.74.2:45557): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
W20260923 23:09:10.631309 21948 leader_election.cc:341] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73a1079bf814192bf19dd63d26c7222 (127.5.74.3:32769): Not found: Tablet not found: 44bc3b44e5d84c4ba9114797c3b9f113
I20260923 23:09:10.631415 21948 leader_election.cc:302] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ab08a72408430fa4497a80180672f2; no voters: 9717ae4977444b91b74514ebb3e335ca, d73a1079bf814192bf19dd63d26c7222
I20260923 23:09:10.631492 22368 raft_consensus.cc:2726] T 44bc3b44e5d84c4ba9114797c3b9f113 P 22ab08a72408430fa4497a80180672f2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:09:10.793990 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:10.793701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:56315)
0923 23:09:10.793890 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:10.793940 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
I20260923 23:09:10.872115 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:10.871851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:56557)
0923 23:09:10.872011 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:10.872064 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260923 23:09:11.007081 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:11.006798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:52791)
0923 23:09:11.006984 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:11.007022 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260923 23:09:11.794868 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:11.794637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:59081)
0923 23:09:11.794776 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:11.794820 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260923 23:09:11.873054 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:11.872782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:44059)
0923 23:09:11.872955 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:11.873003 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260923 23:09:12.007966 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:12.007695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:44087)
0923 23:09:12.007852 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:12.007916 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260923 23:09:12.795930 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:12.795609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:40507)
0923 23:09:12.795792 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:12.795879 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260923 23:09:12.874017 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:12.873735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:47219)
0923 23:09:12.873918 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:12.873965 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260923 23:09:13.008926 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:13.008634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:36441)
0923 23:09:13.008779 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:13.008855 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260923 23:09:13.797111 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:13.796841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:59737)
0923 23:09:13.797007 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:13.797059 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260923 23:09:13.875087 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:13.874718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:47813)
0923 23:09:13.874990 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:13.875037 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":74,"threads_started":1}
I20260923 23:09:14.009778 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:14.009494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:48405)
0923 23:09:14.009662 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:14.009716 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260923 23:09:14.798045 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:14.797793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.1:40431)
0923 23:09:14.797960 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:14.797998 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260923 23:09:14.876142 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:14.875850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.2:51657)
0923 23:09:14.876019 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:14.876091 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
I20260923 23:09:15.005888 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 21897
I20260923 23:09:15.010737 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:15.010513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37359 (local address 127.5.74.3:40487)
0923 23:09:15.010652 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:15.010689 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.62:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260923 23:09:15.011204 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 22064
I20260923 23:09:15.015903 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 22198
2026-09-23T23:09:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58939 ms)
[----------] 30 tests from SecurityITest (240386 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:15 dist-test-slave-j4c5 krb5kdc[22386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:15 dist-test-slave-j4c5 krb5kdc[22386](info): set up 2 sockets
Sep 23 23:09:15 dist-test-slave-j4c5 krb5kdc[22386](info): commencing operation
krb5kdc: starting...
W20260923 23:09:16.071913 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:16 dist-test-slave-j4c5 krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:16.209478 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35043
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:35043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:16.302367 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:16.302640 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:16.302668 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:16.304206 22399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:16.304275 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:16.304291 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:16.304306 22399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:16.304319 22399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:16.306175 22399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:35043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:16.306507 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:16.306782 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:16.312889 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:16.312955 22399 server_base.cc:1034] running on GCE node
W20260923 23:09:16.312847 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:16.312861 22404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:16.313421 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:16.313686 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:16.314855 22399 hybrid_clock.cc:648] HybridClock initialized: now 1790204956314825 us; error 54 us; skew 500 ppm
Sep 23 23:09:16 dist-test-slave-j4c5 krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:16.317464 22399 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:16.317857 22399 webserver.cc:460] Webserver started at http://127.5.74.62:37511/ using document root <none> and password file <none>
I20260923 23:09:16.318073 22399 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:16.318115 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:16.318209 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:16.319252 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "cb3432e4c17f49deb66604936625b07b"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "db017724e11128cc42ca96563dad2e6d"
server_key_iv: "4d668696e130907485ee61a7405bff85"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.319552 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "cb3432e4c17f49deb66604936625b07b"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "db017724e11128cc42ca96563dad2e6d"
server_key_iv: "4d668696e130907485ee61a7405bff85"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.320922 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:09:16.321709 22414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.321977 22399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:16.322077 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "cb3432e4c17f49deb66604936625b07b"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "db017724e11128cc42ca96563dad2e6d"
server_key_iv: "4d668696e130907485ee61a7405bff85"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.322162 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:16.344470 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:16.347079 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:16.347266 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:16.351706 22399 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:35043
I20260923 23:09:16.351809 22466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:35043 every 8 connection(s)
I20260923 23:09:16.352241 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:16.353220 22467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:16.355652 22467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b: Bootstrap starting.
I20260923 23:09:16.356007 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 22399
I20260923 23:09:16.356278 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:16.356534 5416 external_mini_cluster.cc:1428] Setting key f12b5d0ecb3b02e668e0bc7c17870447
I20260923 23:09:16.356642 22467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:16.357026 22467 log.cc:824] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:16.357961 22467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b: No bootstrap required, opened a new log
I20260923 23:09:16.360060 22467 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3432e4c17f49deb66604936625b07b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35043 } }
I20260923 23:09:16.360292 22467 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:16.360344 22467 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3432e4c17f49deb66604936625b07b, State: Initialized, Role: FOLLOWER
I20260923 23:09:16.360530 22467 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3432e4c17f49deb66604936625b07b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35043 } }
I20260923 23:09:16.360631 22467 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:16.360675 22467 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:16.360725 22467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:16.361342 22467 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3432e4c17f49deb66604936625b07b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35043 } }
I20260923 23:09:16.361476 22467 leader_election.cc:302] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb3432e4c17f49deb66604936625b07b; no voters:
I20260923 23:09:16.361676 22467 leader_election.cc:288] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:16.361706 22472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:16.361883 22472 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [term 1 LEADER]: Becoming Leader. State: Replica: cb3432e4c17f49deb66604936625b07b, State: Running, Role: LEADER
I20260923 23:09:16.361979 22472 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3432e4c17f49deb66604936625b07b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35043 } }
I20260923 23:09:16.362010 22467 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:16.362314 22474 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb3432e4c17f49deb66604936625b07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3432e4c17f49deb66604936625b07b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35043 } } }
I20260923 23:09:16.362476 22474 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:16.362319 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb3432e4c17f49deb66604936625b07b. Latest consensus state: current_term: 1 leader_uuid: "cb3432e4c17f49deb66604936625b07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3432e4c17f49deb66604936625b07b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35043 } } }
I20260923 23:09:16.362787 22480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:16.362902 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:16.363456 22480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 23:09:16 dist-test-slave-j4c5 krb5kdc[22386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:16.365413 22480 catalog_manager.cc:1347] Generated new cluster ID: 75d6c376fe75493a8373763415865945
I20260923 23:09:16.365481 22480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:16.368842 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.357285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.5.74.62:35043)
0923 23:09:16.357468 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:16.357478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:16.357517 (+ 39us) server_negotiation.cc:407] Connection header received
0923 23:09:16.358032 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:16.358042 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:16.358186 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:16.358381 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:16.359093 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.360054 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.361062 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.361527 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.361664 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.364643 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:16.364648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:16.364648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:16.364661 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:16.366759 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.367258 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.367261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.367262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.367376 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.367599 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.367600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.367600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.367922 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:16.368080 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:16.368446 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 23:09:16.368611 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260923 23:09:16.377848 22480 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:16.378482 22480 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:16.384945 22480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b: Generated new TSK 0
I20260923 23:09:16.385253 22480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:09:16.419638 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35043
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:16.510475 22493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:16.510754 22493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:16.510799 22493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:16.512038 22493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:16.512113 22493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:16.512161 22493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:09:16.513747 22493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:16.514031 22493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:16.514281 22493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:16.519846 22501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:16.519829 22498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:16.519876 22499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:16.520051 22493 server_base.cc:1034] running on GCE node
I20260923 23:09:16.520354 22493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:16.520581 22493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:16.521729 22493 hybrid_clock.cc:648] HybridClock initialized: now 1790204956521674 us; error 69 us; skew 500 ppm
Sep 23 23:09:16 dist-test-slave-j4c5 krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:16.524529 22493 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:09:16.525012 22493 webserver.cc:460] Webserver started at http://127.5.74.1:33509/ using document root <none> and password file <none>
I20260923 23:09:16.525290 22493 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:16.525341 22493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:16.525514 22493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:16.526640 22493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "84c4d9ac8973457dba2946bae3c6b167"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "37414009ab5cf8a9c9a6ea0fa750a6ea"
server_key_iv: "c2c0d4d765816ab50093fd1c2a77224e"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.527024 22493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "84c4d9ac8973457dba2946bae3c6b167"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "37414009ab5cf8a9c9a6ea0fa750a6ea"
server_key_iv: "c2c0d4d765816ab50093fd1c2a77224e"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.528589 22493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:09:16.529752 22508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.529991 22493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:16.530108 22493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "84c4d9ac8973457dba2946bae3c6b167"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "37414009ab5cf8a9c9a6ea0fa750a6ea"
server_key_iv: "c2c0d4d765816ab50093fd1c2a77224e"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.530210 22493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:16.552232 22493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:16.554898 22493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:16.555033 22493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:16.555315 22493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:16.555656 22493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:16.555692 22493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.555764 22493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:16.555804 22493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.562176 22493 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:35941
I20260923 23:09:16.562252 22621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:35941 every 8 connection(s)
I20260923 23:09:16.562609 22493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:09:16.566123 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 22493
I20260923 23:09:16.566252 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:09:16.566452 5416 external_mini_cluster.cc:1428] Setting key 1d6b6a238176d283e38cc0258d7a8cc0
Sep 23 23:09:16 dist-test-slave-j4c5 krb5kdc[22386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
I20260923 23:09:16.579631 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.563497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:38871 (local address 127.5.74.62:35043)
0923 23:09:16.563552 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:16.563554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:16.563709 (+ 155us) server_negotiation.cc:407] Connection header received
0923 23:09:16.564450 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:16.564452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:16.564465 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:16.564517 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:16.565547 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.566195 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.567523 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.567861 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.567951 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.570879 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:16.570881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:16.570881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:16.570891 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:16.577621 (+ 6730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.578398 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.578401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.578402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.578477 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.578776 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.578778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.578778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.578915 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:16.579030 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:16.579405 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 23:09:16.579511 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:09:16.579707 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.563378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35043 (local address 127.5.74.1:38871)
0923 23:09:16.563556 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:16.563605 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:16.564340 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:16.564593 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:16.564597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:16.565209 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:16.565434 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.565439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.566332 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.566333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.567433 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.567436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.567980 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.567981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.568082 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.568908 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:16.568921 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:16.570767 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:16.577758 (+ 6991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:16.577773 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:16.577777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:16.578250 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:16.578561 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:16.578562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:16.578562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:16.578681 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:16.579024 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:16.579029 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:16.579209 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 23:09:16.579310 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 23:09:16.579486 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260923 23:09:16.580642 22622 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35043
I20260923 23:09:16.580775 22622 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:16.580981 22622 heartbeater.cc:511] Master 127.5.74.62:35043 requested a full tablet report, sending...
I20260923 23:09:16.581576 22431 ts_manager.cc:194] Registered new tserver with Master: 84c4d9ac8973457dba2946bae3c6b167 (127.5.74.1:35941)
I20260923 23:09:16.582211 22431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:38871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:09:16.596450 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35043
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:16.684597 22627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:16.684829 22627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:16.684855 22627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:16.686020 22627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:16.686064 22627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:16.686091 22627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:09:16.687517 22627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:16.687733 22627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:16.687925 22627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:16.693073 22632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:16.693073 22635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:16.693094 22633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:16.693125 22627 server_base.cc:1034] running on GCE node
I20260923 23:09:16.693511 22627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:16.693805 22627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:16.694983 22627 hybrid_clock.cc:648] HybridClock initialized: now 1790204956694958 us; error 42 us; skew 500 ppm
Sep 23 23:09:16 dist-test-slave-j4c5 krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:16.697880 22627 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:09:16.698375 22627 webserver.cc:460] Webserver started at http://127.5.74.2:34375/ using document root <none> and password file <none>
I20260923 23:09:16.698616 22627 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:16.698691 22627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:16.698851 22627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:16.699980 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "955f268e02de4267ab61ce82c8063722"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "563d636fa0cf18d714b669ba77ea11be"
server_key_iv: "879aa7ebcb599b5f8636c45fe599921c"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.700407 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "955f268e02de4267ab61ce82c8063722"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "563d636fa0cf18d714b669ba77ea11be"
server_key_iv: "879aa7ebcb599b5f8636c45fe599921c"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.701815 22627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:16.702781 22642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.703081 22627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:16.703187 22627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "955f268e02de4267ab61ce82c8063722"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "563d636fa0cf18d714b669ba77ea11be"
server_key_iv: "879aa7ebcb599b5f8636c45fe599921c"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.703274 22627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:16.717268 22627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:16.719462 22627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:16.719635 22627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:16.719890 22627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:16.720294 22627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:16.720350 22627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.720392 22627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:16.720418 22627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.727435 22627 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:41547
I20260923 23:09:16.727488 22755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:41547 every 8 connection(s)
I20260923 23:09:16.727873 22627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:09:16.733174 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 22627
I20260923 23:09:16.733289 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:09:16.733515 5416 external_mini_cluster.cc:1428] Setting key 7c1749458ae532fd3e9c43905dc03b94
Sep 23 23:09:16 dist-test-slave-j4c5 krb5kdc[22386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:16.738849 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.728781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:54849 (local address 127.5.74.62:35043)
0923 23:09:16.728843 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:16.728848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:16.729065 (+ 217us) server_negotiation.cc:407] Connection header received
0923 23:09:16.729514 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:16.729515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:16.729526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:16.729596 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:16.730540 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.731076 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.732148 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.732361 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.732427 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.735401 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:16.735403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:16.735404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:16.735413 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:16.737200 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.737746 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.737747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.737748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.737811 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.738117 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.738118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.738118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.738235 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:16.738293 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:16.738688 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 23:09:16.738772 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:09:16.739161 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.728740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35043 (local address 127.5.74.2:54849)
0923 23:09:16.728909 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:16.728951 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:16.729429 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:16.729646 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:16.729649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:16.730235 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:16.730435 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.730440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.731172 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.731173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.731990 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.731992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.732453 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.732454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.732560 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.733206 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:16.733215 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:16.735293 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:16.737297 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:16.737307 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:16.737309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:16.737660 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:16.737884 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:16.737885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:16.737886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:16.738041 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:16.738311 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:16.738318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:16.738517 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 23:09:16.738647 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 23:09:16.738893 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260923 23:09:16.739728 22756 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35043
I20260923 23:09:16.739828 22756 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:16.740031 22756 heartbeater.cc:511] Master 127.5.74.62:35043 requested a full tablet report, sending...
I20260923 23:09:16.740504 22431 ts_manager.cc:194] Registered new tserver with Master: 955f268e02de4267ab61ce82c8063722 (127.5.74.2:41547)
I20260923 23:09:16.740890 22431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:54849
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:09:16.763542 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35043
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:16.849231 22761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:16.849454 22761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:16.849481 22761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:16.850731 22761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:16.850775 22761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:16.850803 22761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:09:16.852315 22761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:35043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:16.852560 22761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:16.852819 22761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:16.858237 22766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:16.858174 22767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:16.858162 22769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:16.858330 22761 server_base.cc:1034] running on GCE node
I20260923 23:09:16.858690 22761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:16.858956 22761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:16.860133 22761 hybrid_clock.cc:648] HybridClock initialized: now 1790204956860113 us; error 41 us; skew 500 ppm
Sep 23 23:09:16 dist-test-slave-j4c5 krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:16.863125 22761 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:09:16.863768 22761 webserver.cc:460] Webserver started at http://127.5.74.3:44779/ using document root <none> and password file <none>
I20260923 23:09:16.864007 22761 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:16.864080 22761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:16.864269 22761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:16.865602 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "062ab646574c4613a2548b551b116a94"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "ec17e4925156b1b45fcaa4b9f65396a7"
server_key_iv: "7fdf13202c64ca2a0c89300e7a290a2a"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.866007 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "062ab646574c4613a2548b551b116a94"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "ec17e4925156b1b45fcaa4b9f65396a7"
server_key_iv: "7fdf13202c64ca2a0c89300e7a290a2a"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.867568 22761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260923 23:09:16.868584 22776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.868782 22761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:16.868877 22761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "062ab646574c4613a2548b551b116a94"
format_stamp: "Formatted at 2026-09-23 23:09:16 on dist-test-slave-j4c5"
server_key: "ec17e4925156b1b45fcaa4b9f65396a7"
server_key_iv: "7fdf13202c64ca2a0c89300e7a290a2a"
server_key_version: "encryptionkey@0"
I20260923 23:09:16.868940 22761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:16.881457 22761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:16.884356 22761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:16.884547 22761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:16.884817 22761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:16.885211 22761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:16.885247 22761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.885272 22761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:16.885283 22761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:16.891309 22761 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:39095
I20260923 23:09:16.891393 22889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:39095 every 8 connection(s)
I20260923 23:09:16.891731 22761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:09:16.899067 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 22761
I20260923 23:09:16.899168 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
Sep 23 23:09:16 dist-test-slave-j4c5 krb5kdc[22386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:16.899392 5416 external_mini_cluster.cc:1428] Setting key c63dceb87b7c9b9e75e08e93dc79bc8d
I20260923 23:09:16.903733 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.892546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:48061 (local address 127.5.74.62:35043)
0923 23:09:16.892597 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:16.892599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:16.892828 (+ 229us) server_negotiation.cc:407] Connection header received
0923 23:09:16.893412 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:16.893414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:16.893427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:16.893490 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:16.894787 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.895314 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.896546 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.896750 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.896859 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.900069 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:16.900073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:16.900074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:16.900093 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:16.901980 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.902558 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.902561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.902561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.902632 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.902948 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.902949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.902950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.903089 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:16.903179 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:16.903505 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 23:09:16.903607 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:09:16.903997 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.892504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:35043 (local address 127.5.74.3:48061)
0923 23:09:16.892699 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:16.892736 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:16.893274 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:16.893561 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:16.893565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:16.894395 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:16.894603 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.894609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.895425 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.895426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.896431 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.896435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.896870 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.896871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.897008 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.897984 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:16.897994 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:16.899809 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:16.902112 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:16.902123 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:16.902126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:16.902431 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:16.902722 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:16.902723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:16.902723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:16.902864 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:16.903168 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:16.903176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:16.903350 (+ 174us) client_negotiation.cc:769] Sending connection context
0923 23:09:16.903483 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 23:09:16.903681 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260923 23:09:16.904577 22890 heartbeater.cc:347] Connected to a master server at 127.5.74.62:35043
I20260923 23:09:16.904692 22890 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:16.904915 22890 heartbeater.cc:511] Master 127.5.74.62:35043 requested a full tablet report, sending...
I20260923 23:09:16.905444 22431 ts_manager.cc:194] Registered new tserver with Master: 062ab646574c4613a2548b551b116a94 (127.5.74.3:39095)
I20260923 23:09:16.905864 22431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:48061
I20260923 23:09:16.912042 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:09:16.920033 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.913504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.5.74.62:35043)
0923 23:09:16.913574 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:16.913576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:16.913705 (+ 129us) server_negotiation.cc:407] Connection header received
0923 23:09:16.913733 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:16.913734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:16.913746 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:16.913812 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:16.914367 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.914924 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.915716 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.915929 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.915989 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.917342 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:16.917344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:16.917344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:16.917351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:16.918782 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.919155 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.919156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.919157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.919248 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:16.919466 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:16.919467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:16.919468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:16.919637 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:16.919724 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:16.919874 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 23:09:16.919936 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:09:16.921727 22431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:09:16.923532 22431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:09:16.931432 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.926471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:39095 (local address 127.0.0.1:51048)
0923 23:09:16.926823 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:16.926874 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:16.926959 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:16.927448 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:16.927449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:16.927451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:16.927513 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.927514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.928543 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.928544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.929951 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.929953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.931015 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.931016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.931122 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.931141 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 23:09:16.931247 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 23:09:16.931335 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":27,"thread_start_us":108,"threads_started":1}
I20260923 23:09:16.931448 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.926586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.5.74.3:39095)
0923 23:09:16.926749 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:16.926754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:16.927202 (+ 448us) server_negotiation.cc:407] Connection header received
0923 23:09:16.927252 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:16.927254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:16.927273 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:16.927363 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:16.927572 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.928416 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.930052 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.930912 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.931004 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.931073 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:16.931275 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 23:09:16.931334 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260923 23:09:16.931622 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.926663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.5.74.1:35941)
0923 23:09:16.926859 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:16.926863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:16.927137 (+ 274us) server_negotiation.cc:407] Connection header received
0923 23:09:16.927192 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:16.927195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:16.927212 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:16.927291 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:16.927567 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.928839 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.930274 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.931066 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.931139 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.931195 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:16.931473 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 23:09:16.931531 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260923 23:09:16.931891 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.926521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35941 (local address 127.0.0.1:44112)
0923 23:09:16.926987 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:16.927000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:16.927066 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:16.927386 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:16.927387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:16.927389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:16.927441 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.927445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.929006 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.929007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.930175 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.930176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.931142 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.931143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.931222 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.931227 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:09:16.931284 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 23:09:16.931771 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260923 23:09:16.932415 22824 tablet_service.cc:1467] Processing CreateTablet for tablet 63dba3318c4f45998aa71e19283c32b7 (DEFAULT_TABLE table=test-table [id=dbe084b15d374c8e9981e9cfdb5c3d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:16.932837 22824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63dba3318c4f45998aa71e19283c32b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:16.932919 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.926735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:41547 (local address 127.0.0.1:56560)
0923 23:09:16.927071 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:16.927081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:16.927123 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:16.927330 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:16.927334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:16.927337 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:16.927425 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.927428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.930077 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.930077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.931013 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:16.931015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.932631 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.932632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:16.932706 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.932712 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 23:09:16.932770 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:09:16.932818 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260923 23:09:16.932891 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 63dba3318c4f45998aa71e19283c32b7 (DEFAULT_TABLE table=test-table [id=dbe084b15d374c8e9981e9cfdb5c3d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:16.933151 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:16.926951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.5.74.2:41547)
0923 23:09:16.927160 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:16.927164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:16.927174 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:09:16.927212 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:16.927215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:16.927230 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:16.927319 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:16.927517 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.928836 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.931125 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:16.932530 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:16.932679 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:16.932745 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:16.932779 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 23:09:16.932827 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260923 23:09:16.933313 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63dba3318c4f45998aa71e19283c32b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:16.934002 22690 tablet_service.cc:1467] Processing CreateTablet for tablet 63dba3318c4f45998aa71e19283c32b7 (DEFAULT_TABLE table=test-table [id=dbe084b15d374c8e9981e9cfdb5c3d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:16.934358 22690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63dba3318c4f45998aa71e19283c32b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:16.935801 22907 tablet_bootstrap.cc:492] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: Bootstrap starting.
I20260923 23:09:16.936545 22907 tablet_bootstrap.cc:654] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:16.937031 22908 tablet_bootstrap.cc:492] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167: Bootstrap starting.
I20260923 23:09:16.937125 22907 log.cc:824] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:16.937700 22908 tablet_bootstrap.cc:654] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:16.938270 22907 tablet_bootstrap.cc:492] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: No bootstrap required, opened a new log
I20260923 23:09:16.938301 22908 log.cc:824] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:16.938349 22907 ts_tablet_manager.cc:1397] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:09:16.938640 22910 tablet_bootstrap.cc:492] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: Bootstrap starting.
I20260923 23:09:16.939078 22908 tablet_bootstrap.cc:492] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167: No bootstrap required, opened a new log
I20260923 23:09:16.939154 22908 ts_tablet_manager.cc:1397] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:09:16.940474 22907 raft_consensus.cc:348] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } }
I20260923 23:09:16.940680 22907 raft_consensus.cc:374] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:16.940714 22907 raft_consensus.cc:729] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062ab646574c4613a2548b551b116a94, State: Initialized, Role: FOLLOWER
I20260923 23:09:16.940850 22910 tablet_bootstrap.cc:654] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:16.940881 22907 consensus_queue.cc:260] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } }
I20260923 23:09:16.941181 22907 ts_tablet_manager.cc:1428] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:09:16.941344 22890 heartbeater.cc:503] Master 127.5.74.62:35043 was elected leader, sending a full tablet report...
I20260923 23:09:16.941231 22908 raft_consensus.cc:348] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } }
I20260923 23:09:16.941404 22908 raft_consensus.cc:374] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:16.941447 22908 raft_consensus.cc:729] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4d9ac8973457dba2946bae3c6b167, State: Initialized, Role: FOLLOWER
I20260923 23:09:16.941736 22908 consensus_queue.cc:260] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } }
I20260923 23:09:16.942035 22910 log.cc:824] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:16.942124 22908 ts_tablet_manager.cc:1428] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:09:16.942168 22622 heartbeater.cc:503] Master 127.5.74.62:35043 was elected leader, sending a full tablet report...
I20260923 23:09:16.943027 22910 tablet_bootstrap.cc:492] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: No bootstrap required, opened a new log
I20260923 23:09:16.943122 22910 ts_tablet_manager.cc:1397] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260923 23:09:16.945013 22910 raft_consensus.cc:348] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } }
I20260923 23:09:16.945166 22910 raft_consensus.cc:374] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:16.945190 22910 raft_consensus.cc:729] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955f268e02de4267ab61ce82c8063722, State: Initialized, Role: FOLLOWER
I20260923 23:09:16.945312 22910 consensus_queue.cc:260] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } }
I20260923 23:09:16.945627 22910 ts_tablet_manager.cc:1428] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 23:09:16.945674 22756 heartbeater.cc:503] Master 127.5.74.62:35043 was elected leader, sending a full tablet report...
W20260923 23:09:16.978543 22757 tablet.cc:2369] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:09:17.063428 22623 tablet.cc:2369] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:09:17.082415 22913 raft_consensus.cc:482] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:17.082566 22913 raft_consensus.cc:504] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } }
I20260923 23:09:17.082870 22913 leader_election.cc:288] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 062ab646574c4613a2548b551b116a94 (127.5.74.3:39095), 955f268e02de4267ab61ce82c8063722 (127.5.74.2:41547)
I20260923 23:09:17.087589 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.083214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:45143 (local address 127.5.74.3:39095)
0923 23:09:17.083301 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.083325 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.083380 (+ 55us) server_negotiation.cc:407] Connection header received
0923 23:09:17.083414 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.083415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.083427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.083490 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:17.083733 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.084539 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.086482 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.087082 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.087175 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.087188 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.087458 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.087511 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 23:09:17.087589 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.083450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:60529 (local address 127.5.74.2:41547)
0923 23:09:17.083500 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.083502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.083511 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:09:17.083573 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.083574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.083583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.083768 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:17.083917 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.084653 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.086482 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.087112 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.087177 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.087188 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.087458 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.087503 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 23:09:17.087633 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.083326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:41547 (local address 127.5.74.1:60529)
0923 23:09:17.083442 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.083449 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.083515 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.083788 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.083788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.083790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:17.083847 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.083849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.084755 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.084755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.086356 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.086360 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.087218 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.087219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.087355 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.087362 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.087439 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.087506 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":144,"thread_start_us":50,"threads_started":1}
I20260923 23:09:17.087633 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.083103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:39095 (local address 127.5.74.1:45143)
0923 23:09:17.083252 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.083266 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.083323 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.083511 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.083512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.083515 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:17.083602 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.083609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.084685 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.084721 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.086356 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.086360 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.087216 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.087218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.087355 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.087361 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.087439 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.087507 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":114,"thread_start_us":104,"threads_started":1}
I20260923 23:09:17.088056 22710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63dba3318c4f45998aa71e19283c32b7" candidate_uuid: "84c4d9ac8973457dba2946bae3c6b167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "955f268e02de4267ab61ce82c8063722" is_pre_election: true
I20260923 23:09:17.088256 22710 raft_consensus.cc:2454] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c4d9ac8973457dba2946bae3c6b167 in term 0.
I20260923 23:09:17.088519 22511 leader_election.cc:302] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4d9ac8973457dba2946bae3c6b167, 955f268e02de4267ab61ce82c8063722; no voters:
I20260923 23:09:17.088738 22844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63dba3318c4f45998aa71e19283c32b7" candidate_uuid: "84c4d9ac8973457dba2946bae3c6b167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062ab646574c4613a2548b551b116a94" is_pre_election: true
I20260923 23:09:17.088790 22913 raft_consensus.cc:2781] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:09:17.088896 22913 raft_consensus.cc:482] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:09:17.088936 22913 raft_consensus.cc:3037] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:17.089054 22844 raft_consensus.cc:2454] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c4d9ac8973457dba2946bae3c6b167 in term 0.
I20260923 23:09:17.089840 22913 raft_consensus.cc:504] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } }
I20260923 23:09:17.089998 22913 leader_election.cc:288] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [CANDIDATE]: Term 1 election: Requested vote from peers 062ab646574c4613a2548b551b116a94 (127.5.74.3:39095), 955f268e02de4267ab61ce82c8063722 (127.5.74.2:41547)
I20260923 23:09:17.090227 22844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63dba3318c4f45998aa71e19283c32b7" candidate_uuid: "84c4d9ac8973457dba2946bae3c6b167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062ab646574c4613a2548b551b116a94"
I20260923 23:09:17.090279 22710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63dba3318c4f45998aa71e19283c32b7" candidate_uuid: "84c4d9ac8973457dba2946bae3c6b167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "955f268e02de4267ab61ce82c8063722"
I20260923 23:09:17.090332 22844 raft_consensus.cc:3037] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:17.090365 22710 raft_consensus.cc:3037] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:17.091500 22710 raft_consensus.cc:2454] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4d9ac8973457dba2946bae3c6b167 in term 1.
I20260923 23:09:17.091500 22844 raft_consensus.cc:2454] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4d9ac8973457dba2946bae3c6b167 in term 1.
I20260923 23:09:17.091732 22511 leader_election.cc:302] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062ab646574c4613a2548b551b116a94, 84c4d9ac8973457dba2946bae3c6b167; no voters:
I20260923 23:09:17.091849 22913 raft_consensus.cc:2781] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:17.092034 22913 raft_consensus.cc:686] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [term 1 LEADER]: Becoming Leader. State: Replica: 84c4d9ac8973457dba2946bae3c6b167, State: Running, Role: LEADER
I20260923 23:09:17.092161 22913 consensus_queue.cc:237] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } }
I20260923 23:09:17.093288 22430 catalog_manager.cc:5495] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c4d9ac8973457dba2946bae3c6b167 (127.5.74.1). New cstate: current_term: 1 leader_uuid: "84c4d9ac8973457dba2946bae3c6b167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c4d9ac8973457dba2946bae3c6b167" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35941 } health_report { overall_health: HEALTHY } } }
I20260923 23:09:17.118281 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.115399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.5.74.1:35941)
0923 23:09:17.115474 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.115476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.115510 (+ 34us) server_negotiation.cc:407] Connection header received
0923 23:09:17.115548 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.115549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.115570 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.115626 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:17.115827 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.116417 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.117355 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.117554 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.117618 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.117828 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:17.117946 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:17.118029 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.118124 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.118144 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 23:09:17.120478 22844 raft_consensus.cc:1261] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [term 1 FOLLOWER]: Refusing update from remote peer 84c4d9ac8973457dba2946bae3c6b167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:17.120482 22710 raft_consensus.cc:1261] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [term 1 FOLLOWER]: Refusing update from remote peer 84c4d9ac8973457dba2946bae3c6b167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:17.120913 22913 consensus_queue.cc:1035] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "062ab646574c4613a2548b551b116a94" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:17.121177 22918 consensus_queue.cc:1035] T 63dba3318c4f45998aa71e19283c32b7 P 84c4d9ac8973457dba2946bae3c6b167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "955f268e02de4267ab61ce82c8063722" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:17.124872 22923 mvcc.cc:205] Tried to move back new op lower bound from 7332679504361930752 to 7332679504250392576. Current Snapshot: MvccSnapshot[applied={T|T < 7332679504361930752 or (T in {7332679504361930752})}]
I20260923 23:09:17.127743 22430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:17.127866 22430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:17.128858 22430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb3432e4c17f49deb66604936625b07b: Sending DeleteTablet for 3 replicas of tablet 63dba3318c4f45998aa71e19283c32b7
I20260923 23:09:17.129575 22690 tablet_service.cc:1514] Processing DeleteTablet for tablet 63dba3318c4f45998aa71e19283c32b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:17 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:56560
I20260923 23:09:17.129776 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 22493
I20260923 23:09:17.130564 22932 tablet_replica.cc:330] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: stopping tablet replica
I20260923 23:09:17.130698 22932 raft_consensus.cc:2229] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:09:17.130825 22932 raft_consensus.cc:2258] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:17.131270 22824 tablet_service.cc:1514] Processing DeleteTablet for tablet 63dba3318c4f45998aa71e19283c32b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:17 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260923 23:09:17.131287 22932 ts_tablet_manager.cc:1905] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:09:17.131536 22933 tablet_replica.cc:330] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: stopping tablet replica
I20260923 23:09:17.131642 22933 raft_consensus.cc:2229] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:09:17.131745 22933 raft_consensus.cc:2258] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:17.132251 22933 ts_tablet_manager.cc:1905] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:09:17.133566 22932 ts_tablet_manager.cc:1918] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:17.133765 22932 log.cc:1196] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/63dba3318c4f45998aa71e19283c32b7
I20260923 23:09:17.134050 22932 ts_tablet_manager.cc:1939] T 63dba3318c4f45998aa71e19283c32b7 P 955f268e02de4267ab61ce82c8063722: Deleting consensus metadata
I20260923 23:09:17.134594 22933 ts_tablet_manager.cc:1918] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:17.134690 22933 log.cc:1196] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/63dba3318c4f45998aa71e19283c32b7
I20260923 23:09:17.135006 22933 ts_tablet_manager.cc:1939] T 63dba3318c4f45998aa71e19283c32b7 P 062ab646574c4613a2548b551b116a94: Deleting consensus metadata
I20260923 23:09:17.135272 22417 catalog_manager.cc:4841] TS 955f268e02de4267ab61ce82c8063722 (127.5.74.2:41547): tablet 63dba3318c4f45998aa71e19283c32b7 (table test-table [id=dbe084b15d374c8e9981e9cfdb5c3d7e]) successfully deleted
I20260923 23:09:17.135434 22417 catalog_manager.cc:4841] TS 062ab646574c4613a2548b551b116a94 (127.5.74.3:39095): tablet 63dba3318c4f45998aa71e19283c32b7 (table test-table [id=dbe084b15d374c8e9981e9cfdb5c3d7e]) successfully deleted
W20260923 23:09:17.137210 22416 connection.cc:535] client connection to 127.5.74.1:35941 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 23:09:17.137313 22416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 23:09:17.137857 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 22627
I20260923 23:09:17.137890 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.137711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35941 (local address 127.0.0.1:44138)
0923 23:09:17.137776 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.137858 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:35941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 23:09:17.139799 22416 catalog_manager.cc:4584] TS 84c4d9ac8973457dba2946bae3c6b167 (127.5.74.1:35941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63dba3318c4f45998aa71e19283c32b7: Network error: Client connection negotiation failed: client connection to 127.5.74.1:35941: connect: Connection refused (error 111)
I20260923 23:09:17.143432 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 22761
I20260923 23:09:17.148454 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 22399
2026-09-23T23:09:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2132 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T23:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:17Z Disabled control of system clock
I20260923 23:09:17.178488 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:40117
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:09:17.264818 22938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:17.265105 22938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:17.265130 22938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:17.266824 22938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:17.266888 22938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:17.266903 22938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:17.266921 22938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:17.266935 22938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:17.268606 22938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:40117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:17.268872 22938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:17.269166 22938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:09:17.274730 22938 server_base.cc:1034] running on GCE node
W20260923 23:09:17.274693 22943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:17.274682 22946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:17.274712 22944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:17.275220 22938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:17.275540 22938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:17.276722 22938 hybrid_clock.cc:648] HybridClock initialized: now 1790204957276700 us; error 43 us; skew 500 ppm
I20260923 23:09:17.278122 22938 webserver.cc:460] Webserver started at http://127.5.74.62:34225/ using document root <none> and password file <none>
I20260923 23:09:17.278403 22938 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:17.278523 22938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:17.278692 22938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:17.279913 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "f683afbd8d6c4edd83f723bfd3e6faa7"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "3bbf1dbee7108e75851fad9cb498425e"
server_key_iv: "a303bd601fb25b2ad51db692786a8e8a"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.280321 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "f683afbd8d6c4edd83f723bfd3e6faa7"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "3bbf1dbee7108e75851fad9cb498425e"
server_key_iv: "a303bd601fb25b2ad51db692786a8e8a"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.281773 22938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:17.282707 22952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.283013 22938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:17.283134 22938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "f683afbd8d6c4edd83f723bfd3e6faa7"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "3bbf1dbee7108e75851fad9cb498425e"
server_key_iv: "a303bd601fb25b2ad51db692786a8e8a"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.283234 22938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:17.297309 22938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:17.297694 22938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:17.297863 22938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:17.302064 22938 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:40117
I20260923 23:09:17.302101 23004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:40117 every 8 connection(s)
I20260923 23:09:17.302485 22938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:17.303715 23005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:17.303831 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 22938
I20260923 23:09:17.303964 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:17.304229 5416 external_mini_cluster.cc:1428] Setting key 11953794cd3aa45faf3587b69eb26874
I20260923 23:09:17.307425 23005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7: Bootstrap starting.
I20260923 23:09:17.308205 23005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:17.308533 23005 log.cc:824] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:17.309262 23005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7: No bootstrap required, opened a new log
I20260923 23:09:17.310549 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.304856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.5.74.62:40117)
0923 23:09:17.305068 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.305075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.305101 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:09:17.305553 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.305560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.305626 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.305752 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:17.306068 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.306984 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.308093 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.308856 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.309032 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.309912 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:17.309916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:17.309916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:17.310091 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:17.310125 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:17.310136 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:17.310218 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.310288 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.310371 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260923 23:09:17.311565 23005 raft_consensus.cc:348] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f683afbd8d6c4edd83f723bfd3e6faa7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40117 } }
I20260923 23:09:17.311730 23005 raft_consensus.cc:374] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:17.311805 23005 raft_consensus.cc:729] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f683afbd8d6c4edd83f723bfd3e6faa7, State: Initialized, Role: FOLLOWER
I20260923 23:09:17.311939 23005 consensus_queue.cc:260] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f683afbd8d6c4edd83f723bfd3e6faa7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40117 } }
I20260923 23:09:17.312036 23005 raft_consensus.cc:388] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:17.312078 23005 raft_consensus.cc:482] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:17.312127 23005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:17.312795 23005 raft_consensus.cc:504] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f683afbd8d6c4edd83f723bfd3e6faa7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40117 } }
I20260923 23:09:17.312918 23005 leader_election.cc:302] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f683afbd8d6c4edd83f723bfd3e6faa7; no voters:
I20260923 23:09:17.313113 23005 leader_election.cc:288] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:17.313268 23010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:17.313443 23010 raft_consensus.cc:686] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [term 1 LEADER]: Becoming Leader. State: Replica: f683afbd8d6c4edd83f723bfd3e6faa7, State: Running, Role: LEADER
I20260923 23:09:17.313505 23005 sys_catalog.cc:564] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:17.313539 23010 consensus_queue.cc:237] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f683afbd8d6c4edd83f723bfd3e6faa7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40117 } }
I20260923 23:09:17.313927 23011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f683afbd8d6c4edd83f723bfd3e6faa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f683afbd8d6c4edd83f723bfd3e6faa7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40117 } } }
I20260923 23:09:17.313931 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f683afbd8d6c4edd83f723bfd3e6faa7. Latest consensus state: current_term: 1 leader_uuid: "f683afbd8d6c4edd83f723bfd3e6faa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f683afbd8d6c4edd83f723bfd3e6faa7" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40117 } } }
I20260923 23:09:17.314040 23011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:17.314044 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7 [sys.catalog]: This master's current role is: LEADER
W20260923 23:09:17.315378 23026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:17.315495 23026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:17.315544 23018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:17.315722 23018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:17.317385 23018 catalog_manager.cc:1347] Generated new cluster ID: 542a48ca299047f78c15e152755f1aca
I20260923 23:09:17.317449 23018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:17.336736 23018 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:17.337500 23018 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:17.349298 23018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7: Generated new TSK 0
I20260923 23:09:17.349557 23018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:17.354877 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40117
--builtin_ntp_servers=127.5.74.20:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:09:17.442786 23029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:17.443063 23029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:17.443104 23029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:17.444437 23029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:17.444511 23029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:17.444555 23029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:09:17.446046 23029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:17.446354 23029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:17.446681 23029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:17.452059 23034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:17.452047 23035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:17.452049 23037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:17.452790 23029 server_base.cc:1034] running on GCE node
I20260923 23:09:17.453056 23029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:17.453326 23029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:17.454535 23029 hybrid_clock.cc:648] HybridClock initialized: now 1790204957454517 us; error 38 us; skew 500 ppm
I20260923 23:09:17.455986 23029 webserver.cc:460] Webserver started at http://127.5.74.1:33089/ using document root <none> and password file <none>
I20260923 23:09:17.456322 23029 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:17.456398 23029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:17.456544 23029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:17.457753 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "5c2d59030975401781078f37ab4b3de2"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "47961d2cd1d4384d98e21c9bed4850d2"
server_key_iv: "fcc2046e7cd4b56288e7c837a1d4bae3"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.458154 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "5c2d59030975401781078f37ab4b3de2"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "47961d2cd1d4384d98e21c9bed4850d2"
server_key_iv: "fcc2046e7cd4b56288e7c837a1d4bae3"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.459561 23029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260923 23:09:17.460403 23043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.460597 23029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:17.460703 23029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "5c2d59030975401781078f37ab4b3de2"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "47961d2cd1d4384d98e21c9bed4850d2"
server_key_iv: "fcc2046e7cd4b56288e7c837a1d4bae3"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.460783 23029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:17.477983 23029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:17.478341 23029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:17.478503 23029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:17.478760 23029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:17.479106 23029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:17.479163 23029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.479206 23029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:17.479231 23029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.485783 23029 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:35423
I20260923 23:09:17.485931 23156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:35423 every 8 connection(s)
I20260923 23:09:17.486284 23029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:09:17.490926 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 23029
I20260923 23:09:17.491060 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:09:17.491640 5416 external_mini_cluster.cc:1428] Setting key 6dbc3706fbfe1267b2c836b1c7627af8
I20260923 23:09:17.492058 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.487066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:46163 (local address 127.5.74.62:40117)
0923 23:09:17.487113 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.487115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.487451 (+ 336us) server_negotiation.cc:407] Connection header received
0923 23:09:17.487997 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.487999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.488016 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.488071 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:17.488802 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.489326 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.490345 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.490562 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.490642 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.491676 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:17.491678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:17.491678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:17.491683 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:17.491729 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:17.491735 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:17.491807 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.491925 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.491962 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:09:17.492359 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.487081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:40117 (local address 127.5.74.1:46163)
0923 23:09:17.487316 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.487353 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.487885 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.488168 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.488178 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.488186 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:17.488700 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.488705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.489431 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.489433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.490246 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.490249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.490655 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.490656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.490829 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.491485 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:09:17.491486 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:17.491530 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:09:17.491565 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:17.491819 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:17.491827 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:17.491831 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.491918 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.492041 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260923 23:09:17.492689 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40117
--builtin_ntp_servers=127.5.74.20:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:09:17.492933 23157 heartbeater.cc:347] Connected to a master server at 127.5.74.62:40117
I20260923 23:09:17.493043 23157 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:17.493232 23157 heartbeater.cc:511] Master 127.5.74.62:40117 requested a full tablet report, sending...
I20260923 23:09:17.493805 22969 ts_manager.cc:194] Registered new tserver with Master: 5c2d59030975401781078f37ab4b3de2 (127.5.74.1:35423)
I20260923 23:09:17.494558 22969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.1:46163
W20260923 23:09:17.580526 23160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:17.580797 23160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:17.580822 23160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:17.582041 23160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:17.582082 23160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:17.582108 23160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:09:17.583447 23160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:17.583877 23160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:17.584219 23160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:17.589867 23168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:17.589908 23166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:17.589867 23165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:17.590197 23160 server_base.cc:1034] running on GCE node
I20260923 23:09:17.590495 23160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:17.590808 23160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:17.591998 23160 hybrid_clock.cc:648] HybridClock initialized: now 1790204957591983 us; error 32 us; skew 500 ppm
I20260923 23:09:17.593421 23160 webserver.cc:460] Webserver started at http://127.5.74.2:35747/ using document root <none> and password file <none>
I20260923 23:09:17.593747 23160 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:17.593823 23160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:17.593981 23160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:17.595099 23160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "86c2fd2ce64547b18b33855135623396"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "baab07623e2e2b386d601b632d1be89e"
server_key_iv: "859328f4053f684c38450c0ca9937b6e"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.595480 23160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "86c2fd2ce64547b18b33855135623396"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "baab07623e2e2b386d601b632d1be89e"
server_key_iv: "859328f4053f684c38450c0ca9937b6e"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.597139 23160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:17.598097 23174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.598398 23160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:17.598527 23160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "86c2fd2ce64547b18b33855135623396"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "baab07623e2e2b386d601b632d1be89e"
server_key_iv: "859328f4053f684c38450c0ca9937b6e"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.598613 23160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:17.618721 23160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:17.619095 23160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:17.619266 23160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:17.619552 23160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:17.619907 23160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:17.619967 23160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.620010 23160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:17.620039 23160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.626255 23160 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:38689
I20260923 23:09:17.626302 23287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:38689 every 8 connection(s)
I20260923 23:09:17.626663 23160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:09:17.629247 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 23160
I20260923 23:09:17.629366 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:09:17.629614 5416 external_mini_cluster.cc:1428] Setting key 90812d4814040112474a31490731c2b4
I20260923 23:09:17.630722 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40117
--builtin_ntp_servers=127.5.74.20:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:09:17.633137 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.627508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:58555 (local address 127.5.74.62:40117)
0923 23:09:17.627581 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.627583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.627782 (+ 199us) server_negotiation.cc:407] Connection header received
0923 23:09:17.628331 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.628332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.628346 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.628420 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:17.629200 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.629977 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.631296 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.631674 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.631755 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.632783 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:17.632785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:17.632785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:17.632790 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:17.632811 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:17.632816 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:17.632892 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.633027 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.633070 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:09:17.633381 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.627435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:40117 (local address 127.5.74.2:58555)
0923 23:09:17.627618 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.627660 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.628178 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.628532 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.628536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.628545 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:17.629063 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.629067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.630094 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.630095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.631202 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.631205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.631779 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.631780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.631909 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.632588 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:09:17.632589 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:17.632632 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:09:17.632667 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:17.632904 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:17.632910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:17.632913 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.632988 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.633133 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260923 23:09:17.633862 23288 heartbeater.cc:347] Connected to a master server at 127.5.74.62:40117
I20260923 23:09:17.633984 23288 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:17.634228 23288 heartbeater.cc:511] Master 127.5.74.62:40117 requested a full tablet report, sending...
I20260923 23:09:17.634618 22969 ts_manager.cc:194] Registered new tserver with Master: 86c2fd2ce64547b18b33855135623396 (127.5.74.2:38689)
I20260923 23:09:17.635026 22969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.2:58555
W20260923 23:09:17.717126 23291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:17.717374 23291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:17.717401 23291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:17.718704 23291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:17.718750 23291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:17.718778 23291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:09:17.720275 23291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:17.720521 23291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:17.720793 23291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:17.726106 23299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:17.726143 23297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:17.726136 23291 server_base.cc:1034] running on GCE node
W20260923 23:09:17.726106 23296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:17.726652 23291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:17.726933 23291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:17.728109 23291 hybrid_clock.cc:648] HybridClock initialized: now 1790204957728098 us; error 34 us; skew 500 ppm
I20260923 23:09:17.729616 23291 webserver.cc:460] Webserver started at http://127.5.74.3:38913/ using document root <none> and password file <none>
I20260923 23:09:17.729904 23291 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:17.729981 23291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:17.730136 23291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:17.731227 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "8a9b2c1b9ec24229953eba84be5c74c4"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "851984d6d2621ce4047050efcab1eaee"
server_key_iv: "ca8e4cce2dc2923a8d92eca22dd0b63b"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.731596 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "8a9b2c1b9ec24229953eba84be5c74c4"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "851984d6d2621ce4047050efcab1eaee"
server_key_iv: "ca8e4cce2dc2923a8d92eca22dd0b63b"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.733052 23291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:17.733954 23305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.734165 23291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:17.734278 23291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "8a9b2c1b9ec24229953eba84be5c74c4"
format_stamp: "Formatted at 2026-09-23 23:09:17 on dist-test-slave-j4c5"
server_key: "851984d6d2621ce4047050efcab1eaee"
server_key_iv: "ca8e4cce2dc2923a8d92eca22dd0b63b"
server_key_version: "encryptionkey@0"
I20260923 23:09:17.734397 23291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:17.744796 23291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:17.745343 23291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:17.745563 23291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:17.745855 23291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:17.746213 23291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:17.746275 23291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.746352 23291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:17.746384 23291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:17.752455 23291 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:33367
I20260923 23:09:17.752679 23418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:33367 every 8 connection(s)
I20260923 23:09:17.752940 23291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:09:17.756911 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 23291
I20260923 23:09:17.757056 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:09:17.757294 5416 external_mini_cluster.cc:1428] Setting key af33aefcf84836ce2e5a7ac5e09bc0c4
I20260923 23:09:17.758901 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.753915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:34955 (local address 127.5.74.62:40117)
0923 23:09:17.754010 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.754012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.754215 (+ 203us) server_negotiation.cc:407] Connection header received
0923 23:09:17.754764 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.754765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.754776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.754837 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:17.755571 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.756106 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.757275 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.757576 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.757649 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.758643 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:17.758645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:17.758646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:17.758649 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:17.758665 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:17.758668 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:17.758714 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.758817 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.758840 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 23:09:17.759119 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.753841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:40117 (local address 127.5.74.3:34955)
0923 23:09:17.754060 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.754097 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.754660 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.754894 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.754898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.754907 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:17.755475 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.755479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.756239 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.756240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.757174 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.757177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.757660 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.757661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.757849 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.758494 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:09:17.758494 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:17.758533 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:09:17.758569 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:17.758737 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:17.758741 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:17.758744 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.758812 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.758903 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260923 23:09:17.759706 23419 heartbeater.cc:347] Connected to a master server at 127.5.74.62:40117
I20260923 23:09:17.759820 23419 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:17.760025 23419 heartbeater.cc:511] Master 127.5.74.62:40117 requested a full tablet report, sending...
I20260923 23:09:17.760496 22969 ts_manager.cc:194] Registered new tserver with Master: 8a9b2c1b9ec24229953eba84be5c74c4 (127.5.74.3:33367)
I20260923 23:09:17.760948 22969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.3:34955
I20260923 23:09:17.769793 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:09:17.774757 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.771358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.5.74.62:40117)
0923 23:09:17.771404 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.771406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.771577 (+ 171us) server_negotiation.cc:407] Connection header received
0923 23:09:17.771606 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.771607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.771618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.771677 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:17.771850 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.772429 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.773437 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.773625 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.773684 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.774413 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:17.774415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:17.774415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:17.774419 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:17.774434 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:17.774438 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:17.774569 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.774658 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.774682 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:09:17.777076 22969 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:09:17.778626 22969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:09:17.785635 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.781274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35423 (local address 127.0.0.1:39950)
0923 23:09:17.781698 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.781729 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.781799 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.782064 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.782067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.782070 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:17.782153 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.782156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.783618 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.783619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.784577 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.784579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.785299 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.785300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.785387 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.785395 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.785475 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.785547 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":25,"spinlock_wait_cycles":896,"thread_start_us":109,"threads_started":1}
I20260923 23:09:17.785686 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.781493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.5.74.2:38689)
0923 23:09:17.781716 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.781723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.781885 (+ 162us) server_negotiation.cc:407] Connection header received
0923 23:09:17.781938 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.781941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.781959 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.782032 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:17.782247 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.782995 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.784347 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.785099 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.785174 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.785262 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.785537 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.785581 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260923 23:09:17.785882 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.781274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:38689 (local address 127.0.0.1:45342)
0923 23:09:17.781681 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.781724 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.781799 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.782104 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.782105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.782107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:17.782171 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.782173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.783090 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.783091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.784259 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.784262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.785296 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.785297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.785378 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.785393 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.785467 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.785537 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260923 23:09:17.786158 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.781500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.5.74.1:35423)
0923 23:09:17.781797 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.781801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.781813 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:09:17.781892 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.781894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.781910 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.781985 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:17.782235 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.783528 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.784661 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.785170 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.785297 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.785389 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.785964 (+ 575us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.786074 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":77,"threads_started":1}
I20260923 23:09:17.786625 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.781535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:33367 (local address 127.0.0.1:50890)
0923 23:09:17.781940 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.781949 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.782000 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.782246 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.782246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.782248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:17.782280 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.782282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.784255 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.784256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.785371 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.785373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.786398 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.786399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.786458 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.786462 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.786523 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.786563 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":75,"threads_started":1}
I20260923 23:09:17.786722 23222 tablet_service.cc:1467] Processing CreateTablet for tablet 0c68e5e4def848c69e7473176f993c4f (DEFAULT_TABLE table=test-table [id=02f79a1ee9384fd2b5488b99b7a07712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:17.786931 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.781622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.5.74.3:33367)
0923 23:09:17.781823 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.781828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.782059 (+ 231us) server_negotiation.cc:407] Connection header received
0923 23:09:17.782103 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.782105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.782119 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.782186 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:17.782361 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.783521 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.785464 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.786214 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.786280 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.786363 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.786805 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.786848 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260923 23:09:17.786904 23091 tablet_service.cc:1467] Processing CreateTablet for tablet 0c68e5e4def848c69e7473176f993c4f (DEFAULT_TABLE table=test-table [id=02f79a1ee9384fd2b5488b99b7a07712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:17.787158 23222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c68e5e4def848c69e7473176f993c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:17.787356 23091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c68e5e4def848c69e7473176f993c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:17.787642 23353 tablet_service.cc:1467] Processing CreateTablet for tablet 0c68e5e4def848c69e7473176f993c4f (DEFAULT_TABLE table=test-table [id=02f79a1ee9384fd2b5488b99b7a07712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:17.787961 23353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c68e5e4def848c69e7473176f993c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:17.790023 23433 tablet_bootstrap.cc:492] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: Bootstrap starting.
I20260923 23:09:17.790719 23433 tablet_bootstrap.cc:654] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:17.790869 23434 tablet_bootstrap.cc:492] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2: Bootstrap starting.
I20260923 23:09:17.791059 23433 log.cc:824] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:17.791579 23434 tablet_bootstrap.cc:654] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:17.791899 23433 tablet_bootstrap.cc:492] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: No bootstrap required, opened a new log
I20260923 23:09:17.791893 23434 log.cc:824] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:17.791966 23433 ts_tablet_manager.cc:1397] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:09:17.792804 23434 tablet_bootstrap.cc:492] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2: No bootstrap required, opened a new log
I20260923 23:09:17.792881 23434 ts_tablet_manager.cc:1397] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:09:17.794633 23433 raft_consensus.cc:348] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } }
I20260923 23:09:17.794838 23433 raft_consensus.cc:374] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:17.794874 23433 raft_consensus.cc:729] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c2fd2ce64547b18b33855135623396, State: Initialized, Role: FOLLOWER
I20260923 23:09:17.795023 23433 consensus_queue.cc:260] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } }
I20260923 23:09:17.795313 23433 ts_tablet_manager.cc:1428] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:09:17.795361 23288 heartbeater.cc:503] Master 127.5.74.62:40117 was elected leader, sending a full tablet report...
I20260923 23:09:17.791512 23436 tablet_bootstrap.cc:492] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: Bootstrap starting.
I20260923 23:09:17.795578 23434 raft_consensus.cc:348] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } }
I20260923 23:09:17.795758 23434 raft_consensus.cc:374] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:17.795796 23434 raft_consensus.cc:729] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2d59030975401781078f37ab4b3de2, State: Initialized, Role: FOLLOWER
I20260923 23:09:17.795909 23434 consensus_queue.cc:260] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } }
I20260923 23:09:17.796190 23436 tablet_bootstrap.cc:654] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:17.796200 23434 ts_tablet_manager.cc:1428] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:09:17.796334 23157 heartbeater.cc:503] Master 127.5.74.62:40117 was elected leader, sending a full tablet report...
I20260923 23:09:17.796550 23436 log.cc:824] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:17.797518 23436 tablet_bootstrap.cc:492] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: No bootstrap required, opened a new log
I20260923 23:09:17.797597 23436 ts_tablet_manager.cc:1397] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260923 23:09:17.799629 23436 raft_consensus.cc:348] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } }
I20260923 23:09:17.799765 23436 raft_consensus.cc:374] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:17.799787 23436 raft_consensus.cc:729] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9b2c1b9ec24229953eba84be5c74c4, State: Initialized, Role: FOLLOWER
I20260923 23:09:17.799947 23436 consensus_queue.cc:260] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } }
I20260923 23:09:17.800261 23436 ts_tablet_manager.cc:1428] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 23:09:17.800335 23419 heartbeater.cc:503] Master 127.5.74.62:40117 was elected leader, sending a full tablet report...
W20260923 23:09:17.877279 23289 tablet.cc:2369] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:09:17.928542 23438 raft_consensus.cc:482] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:17.928709 23438 raft_consensus.cc:504] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } }
I20260923 23:09:17.929149 23438 leader_election.cc:288] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c2d59030975401781078f37ab4b3de2 (127.5.74.1:35423), 8a9b2c1b9ec24229953eba84be5c74c4 (127.5.74.3:33367)
I20260923 23:09:17.932227 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.929481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:46001 (local address 127.5.74.3:33367)
0923 23:09:17.929547 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.929549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.929557 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:09:17.929585 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.929586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.929604 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.929673 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:17.929846 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.930358 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.931304 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.931825 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.931888 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.931898 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.932097 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.932136 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:09:17.932228 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.929337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:33367 (local address 127.5.74.2:46001)
0923 23:09:17.929383 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.929394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.929445 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.929679 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.929680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.929682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:17.929755 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.929756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.930451 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.930452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.931206 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.931208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.931932 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.931943 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.932017 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.932025 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.932092 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.932135 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 23:09:17.932518 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.929421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35423 (local address 127.5.74.2:42435)
0923 23:09:17.929592 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.929601 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:17.929691 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:17.929838 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:17.929839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:17.929840 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:17.929910 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.929912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.930658 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.930663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.931488 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:17.931491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.932190 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.932191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:17.932262 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.932266 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:09:17.932372 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 23:09:17.932426 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260923 23:09:17.932567 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.929505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:42435 (local address 127.5.74.1:35423)
0923 23:09:17.929560 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.929562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.929675 (+ 113us) server_negotiation.cc:407] Connection header received
0923 23:09:17.929748 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.929749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.929768 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.929834 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:17.929968 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.930538 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.931585 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.932086 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.932200 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.932210 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.932455 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.932503 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:09:17.932559 23373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c68e5e4def848c69e7473176f993c4f" candidate_uuid: "86c2fd2ce64547b18b33855135623396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" is_pre_election: true
I20260923 23:09:17.932726 23373 raft_consensus.cc:2454] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86c2fd2ce64547b18b33855135623396 in term 0.
I20260923 23:09:17.932899 23111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c68e5e4def848c69e7473176f993c4f" candidate_uuid: "86c2fd2ce64547b18b33855135623396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2d59030975401781078f37ab4b3de2" is_pre_election: true
I20260923 23:09:17.932981 23178 leader_election.cc:302] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86c2fd2ce64547b18b33855135623396, 8a9b2c1b9ec24229953eba84be5c74c4; no voters:
I20260923 23:09:17.933049 23111 raft_consensus.cc:2454] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86c2fd2ce64547b18b33855135623396 in term 0.
I20260923 23:09:17.933107 23438 raft_consensus.cc:2781] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:09:17.933156 23438 raft_consensus.cc:482] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:09:17.933204 23438 raft_consensus.cc:3037] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:17.933928 23438 raft_consensus.cc:504] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } }
I20260923 23:09:17.934073 23438 leader_election.cc:288] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [CANDIDATE]: Term 1 election: Requested vote from peers 5c2d59030975401781078f37ab4b3de2 (127.5.74.1:35423), 8a9b2c1b9ec24229953eba84be5c74c4 (127.5.74.3:33367)
I20260923 23:09:17.934231 23111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c68e5e4def848c69e7473176f993c4f" candidate_uuid: "86c2fd2ce64547b18b33855135623396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2d59030975401781078f37ab4b3de2"
I20260923 23:09:17.934283 23373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c68e5e4def848c69e7473176f993c4f" candidate_uuid: "86c2fd2ce64547b18b33855135623396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9b2c1b9ec24229953eba84be5c74c4"
I20260923 23:09:17.934316 23111 raft_consensus.cc:3037] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:17.934389 23373 raft_consensus.cc:3037] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:17.935137 23373 raft_consensus.cc:2454] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86c2fd2ce64547b18b33855135623396 in term 1.
I20260923 23:09:17.935138 23111 raft_consensus.cc:2454] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86c2fd2ce64547b18b33855135623396 in term 1.
I20260923 23:09:17.935307 23178 leader_election.cc:302] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86c2fd2ce64547b18b33855135623396, 8a9b2c1b9ec24229953eba84be5c74c4; no voters:
I20260923 23:09:17.935413 23438 raft_consensus.cc:2781] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:17.935559 23438 raft_consensus.cc:686] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 1 LEADER]: Becoming Leader. State: Replica: 86c2fd2ce64547b18b33855135623396, State: Running, Role: LEADER
I20260923 23:09:17.935657 23438 consensus_queue.cc:237] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } }
I20260923 23:09:17.936488 22969 catalog_manager.cc:5495] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86c2fd2ce64547b18b33855135623396 (127.5.74.2). New cstate: current_term: 1 leader_uuid: "86c2fd2ce64547b18b33855135623396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86c2fd2ce64547b18b33855135623396" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 } health_report { overall_health: UNKNOWN } } }
I20260923 23:09:17.972815 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.970240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.5.74.2:38689)
0923 23:09:17.970333 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:17.970335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:17.970343 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:09:17.970421 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:17.970422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:17.970446 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:17.970512 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:17.970623 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.971127 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.971896 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:17.972106 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:17.972248 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:17.972431 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:17.972551 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:17.972640 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:17.972707 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 23:09:17.972728 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 23:09:17.974831 23373 raft_consensus.cc:1261] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [term 1 FOLLOWER]: Refusing update from remote peer 86c2fd2ce64547b18b33855135623396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:17.974846 23111 raft_consensus.cc:1261] T 0c68e5e4def848c69e7473176f993c4f P 5c2d59030975401781078f37ab4b3de2 [term 1 FOLLOWER]: Refusing update from remote peer 86c2fd2ce64547b18b33855135623396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:17.975256 23443 consensus_queue.cc:1035] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9b2c1b9ec24229953eba84be5c74c4" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 33367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:17.975399 23438 consensus_queue.cc:1035] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c2d59030975401781078f37ab4b3de2" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:17.978631 23448 mvcc.cc:205] Tried to move back new op lower bound from 7332679507862020096 to 7332679507705102336. Current Snapshot: MvccSnapshot[applied={T|T < 7332679507862020096}]
I20260923 23:09:17.980728 22968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:17.980844 22968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:17.981705 22968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f683afbd8d6c4edd83f723bfd3e6faa7: Sending DeleteTablet for 3 replicas of tablet 0c68e5e4def848c69e7473176f993c4f
I20260923 23:09:17.982090 23091 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c68e5e4def848c69e7473176f993c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:17 UTC) from {username='slave'} at 127.0.0.1:39950
I20260923 23:09:17.982110 23222 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c68e5e4def848c69e7473176f993c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:17 UTC) from {username='slave'} at 127.0.0.1:45342
I20260923 23:09:17.982350 23353 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c68e5e4def848c69e7473176f993c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:17 UTC) from {username='slave'} at 127.0.0.1:50890
I20260923 23:09:17.982512 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 23029
I20260923 23:09:17.983067 23455 tablet_replica.cc:330] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: stopping tablet replica
I20260923 23:09:17.983173 23455 raft_consensus.cc:2229] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:09:17.983302 23455 raft_consensus.cc:2258] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:17.983788 23455 ts_tablet_manager.cc:1905] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:09:17.986020 23455 ts_tablet_manager.cc:1918] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:17.986099 23455 log.cc:1196] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/0c68e5e4def848c69e7473176f993c4f
I20260923 23:09:17.986361 23455 ts_tablet_manager.cc:1939] T 0c68e5e4def848c69e7473176f993c4f P 86c2fd2ce64547b18b33855135623396: Deleting consensus metadata
I20260923 23:09:17.987110 22956 catalog_manager.cc:4841] TS 86c2fd2ce64547b18b33855135623396 (127.5.74.2:38689): tablet 0c68e5e4def848c69e7473176f993c4f (table test-table [id=02f79a1ee9384fd2b5488b99b7a07712]) successfully deleted
I20260923 23:09:17.988310 23457 tablet_replica.cc:330] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: stopping tablet replica
I20260923 23:09:17.988478 23457 raft_consensus.cc:2229] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:09:17.988631 23457 raft_consensus.cc:2258] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:17.989076 23457 ts_tablet_manager.cc:1905] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 23:09:17.989857 22955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:35423 (error 108)
I20260923 23:09:17.990036 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 23160
I20260923 23:09:17.990973 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:17.990729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35423 (local address 127.0.0.1:39964)
0923 23:09:17.990877 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:17.990929 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:35423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260923 23:09:17.991219 22955 catalog_manager.cc:4584] TS 5c2d59030975401781078f37ab4b3de2 (127.5.74.1:35423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c68e5e4def848c69e7473176f993c4f: Network error: Client connection negotiation failed: client connection to 127.5.74.1:35423: connect: Connection refused (error 111)
I20260923 23:09:17.991276 23457 ts_tablet_manager.cc:1918] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:17.991335 23457 log.cc:1196] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/0c68e5e4def848c69e7473176f993c4f
I20260923 23:09:17.991590 23457 ts_tablet_manager.cc:1939] T 0c68e5e4def848c69e7473176f993c4f P 8a9b2c1b9ec24229953eba84be5c74c4: Deleting consensus metadata
I20260923 23:09:17.992200 22956 catalog_manager.cc:4841] TS 8a9b2c1b9ec24229953eba84be5c74c4 (127.5.74.3:33367): tablet 0c68e5e4def848c69e7473176f993c4f (table test-table [id=02f79a1ee9384fd2b5488b99b7a07712]) successfully deleted
I20260923 23:09:17.995334 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 23291
I20260923 23:09:18.000349 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 22938
2026-09-23T23:09:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (851 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:18 dist-test-slave-j4c5 krb5kdc[23461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:18 dist-test-slave-j4c5 krb5kdc[23461](info): set up 2 sockets
Sep 23 23:09:18 dist-test-slave-j4c5 krb5kdc[23461](info): commencing operation
krb5kdc: starting...
W20260923 23:09:19.056105 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:19 dist-test-slave-j4c5 krb5kdc[23461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:19.188591 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37663
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:19.272522 23474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:19.272821 23474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:19.272863 23474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:19.274906 23474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:19.274994 23474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:19.275018 23474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:19.275039 23474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:19.275069 23474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:19.276851 23474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:37663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:19.277169 23474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:19.277420 23474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:19.282753 23482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:19.282770 23480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:19.282889 23474 server_base.cc:1034] running on GCE node
W20260923 23:09:19.282771 23479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:19.283355 23474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:19.283682 23474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:19.284869 23474 hybrid_clock.cc:648] HybridClock initialized: now 1790204959284846 us; error 39 us; skew 500 ppm
Sep 23 23:09:19 dist-test-slave-j4c5 krb5kdc[23461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204959, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:19.287606 23474 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:19.288196 23474 webserver.cc:460] Webserver started at http://127.5.74.62:39927/ using document root <none> and password file <none>
I20260923 23:09:19.288445 23474 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:19.288517 23474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:19.288668 23474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:19.289780 23474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "e89afc85b1ba4f77848d55b5518f6824"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "ecf275223f90d1ca5c6b5ba1559b29b2"
server_key_iv: "4da76c929126f42b5892e48ced16bb76"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.290162 23474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "e89afc85b1ba4f77848d55b5518f6824"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "ecf275223f90d1ca5c6b5ba1559b29b2"
server_key_iv: "4da76c929126f42b5892e48ced16bb76"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.291563 23474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:19.292483 23489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.292737 23474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:19.292860 23474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "e89afc85b1ba4f77848d55b5518f6824"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "ecf275223f90d1ca5c6b5ba1559b29b2"
server_key_iv: "4da76c929126f42b5892e48ced16bb76"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.292949 23474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:19.304388 23474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:19.307126 23474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:19.307343 23474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:19.311939 23474 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:37663
I20260923 23:09:19.311980 23541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:37663 every 8 connection(s)
I20260923 23:09:19.312423 23474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:19.313416 23542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:19.314304 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 23474
I20260923 23:09:19.314453 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:19.314692 5416 external_mini_cluster.cc:1428] Setting key c6d85f0815bafbe07641718b7fb10398
I20260923 23:09:19.317044 23542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824: Bootstrap starting.
I20260923 23:09:19.317818 23542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:19.318157 23542 log.cc:824] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:19.319111 23542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824: No bootstrap required, opened a new log
I20260923 23:09:19.321452 23542 raft_consensus.cc:348] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89afc85b1ba4f77848d55b5518f6824" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37663 } }
I20260923 23:09:19.321645 23542 raft_consensus.cc:374] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:19.321689 23542 raft_consensus.cc:729] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e89afc85b1ba4f77848d55b5518f6824, State: Initialized, Role: FOLLOWER
I20260923 23:09:19.321854 23542 consensus_queue.cc:260] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89afc85b1ba4f77848d55b5518f6824" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37663 } }
I20260923 23:09:19.321949 23542 raft_consensus.cc:388] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:19.322002 23542 raft_consensus.cc:482] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:19.322060 23542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 23:09:19 dist-test-slave-j4c5 krb5kdc[23461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:19.322960 23542 raft_consensus.cc:504] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89afc85b1ba4f77848d55b5518f6824" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37663 } }
I20260923 23:09:19.323084 23542 leader_election.cc:302] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e89afc85b1ba4f77848d55b5518f6824; no voters:
I20260923 23:09:19.323310 23542 leader_election.cc:288] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:19.323364 23547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:19.323552 23547 raft_consensus.cc:686] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [term 1 LEADER]: Becoming Leader. State: Replica: e89afc85b1ba4f77848d55b5518f6824, State: Running, Role: LEADER
I20260923 23:09:19.323633 23542 sys_catalog.cc:564] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:19.323714 23547 consensus_queue.cc:237] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89afc85b1ba4f77848d55b5518f6824" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37663 } }
I20260923 23:09:19.324355 23548 sys_catalog.cc:455] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e89afc85b1ba4f77848d55b5518f6824. Latest consensus state: current_term: 1 leader_uuid: "e89afc85b1ba4f77848d55b5518f6824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89afc85b1ba4f77848d55b5518f6824" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37663 } } }
I20260923 23:09:19.324862 23548 sys_catalog.cc:458] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:19.324760 23549 sys_catalog.cc:455] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e89afc85b1ba4f77848d55b5518f6824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89afc85b1ba4f77848d55b5518f6824" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 37663 } } }
I20260923 23:09:19.325093 23559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:19.325135 23549 sys_catalog.cc:458] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:19.325666 23559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:19.327029 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.315320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.5.74.62:37663)
0923 23:09:19.315533 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:19.315543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:19.315577 (+ 34us) server_negotiation.cc:407] Connection header received
0923 23:09:19.316138 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:19.316147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:19.316247 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:19.316422 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:19.317058 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.318504 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.319450 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.319879 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.320010 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.322938 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:19.322943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:19.322943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:19.322954 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:19.325296 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.325765 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.325769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.325770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.325872 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.326080 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.326081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.326081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.326384 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:19.326533 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:19.326657 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 23:09:19.326807 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260923 23:09:19.327526 23559 catalog_manager.cc:1347] Generated new cluster ID: ebfc7c05cfc54f1e94fe626f26d0b9a7
I20260923 23:09:19.327569 23559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:19.334143 23559 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:19.334913 23559 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:19.343806 23559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824: Generated new TSK 0
I20260923 23:09:19.344048 23559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:09:19.376084 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37663
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:19.462211 23568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:19.462471 23568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:19.462527 23568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:19.463763 23568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:19.463845 23568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:19.463892 23568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:09:19.465477 23568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:19.465803 23568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:19.466058 23568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:19.471652 23576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:19.471617 23568 server_base.cc:1034] running on GCE node
W20260923 23:09:19.471586 23574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:19.471593 23573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:19.472092 23568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:19.472373 23568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:19.473596 23568 hybrid_clock.cc:648] HybridClock initialized: now 1790204959473575 us; error 35 us; skew 500 ppm
Sep 23 23:09:19 dist-test-slave-j4c5 krb5kdc[23461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204959, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:19.476511 23568 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:09:19.477048 23568 webserver.cc:460] Webserver started at http://127.5.74.1:41473/ using document root <none> and password file <none>
I20260923 23:09:19.477308 23568 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:19.477358 23568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:19.477504 23568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:19.478605 23568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "35659ef7d3ac4d219a536e770d1d5b71"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "38de3aa462a8d117bdeb724e1e044759"
server_key_iv: "b6fee81eaa48b4cc269d998a7901b0b8"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.479040 23568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "35659ef7d3ac4d219a536e770d1d5b71"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "38de3aa462a8d117bdeb724e1e044759"
server_key_iv: "b6fee81eaa48b4cc269d998a7901b0b8"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.480803 23568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:09:19.481774 23583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.482005 23568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:19.482084 23568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "35659ef7d3ac4d219a536e770d1d5b71"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "38de3aa462a8d117bdeb724e1e044759"
server_key_iv: "b6fee81eaa48b4cc269d998a7901b0b8"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.482185 23568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:19.499980 23568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:19.502316 23568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:19.502491 23568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:19.502755 23568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:19.503090 23568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:19.503144 23568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.503180 23568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:19.503206 23568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.509066 23568 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:35899
I20260923 23:09:19.509114 23696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:35899 every 8 connection(s)
I20260923 23:09:19.509442 23568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:09:19.511874 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 23568
I20260923 23:09:19.511994 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:09:19.512259 5416 external_mini_cluster.cc:1428] Setting key 12f4108e4882fb3d97c15864342e6d73
Sep 23 23:09:19 dist-test-slave-j4c5 krb5kdc[23461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204959, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:19.519989 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.510203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:59431 (local address 127.5.74.62:37663)
0923 23:09:19.510291 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:19.510293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:19.510447 (+ 154us) server_negotiation.cc:407] Connection header received
0923 23:09:19.510951 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:19.510952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:19.510963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:19.511047 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:19.512016 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.512850 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.513859 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.514084 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.514164 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.516777 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:19.516778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:19.516778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:19.516784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:19.518471 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.518932 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.518933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.518934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.518998 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.519249 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.519250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.519251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.519407 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:19.519482 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:19.519756 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 23:09:19.519857 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 23:09:19.520021 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.510155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37663 (local address 127.5.74.1:59431)
0923 23:09:19.510317 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:19.510354 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:19.510831 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:19.511102 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:19.511106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:19.511677 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:19.511890 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.511894 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.512939 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.512940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.513754 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.513756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.514174 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.514176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.514303 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.514903 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:19.514911 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:19.516645 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:19.518583 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:19.518589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:19.518592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:19.518850 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:19.519073 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:19.519073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:19.519074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:19.519180 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:19.519477 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:19.519481 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:19.519621 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 23:09:19.519709 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 23:09:19.519842 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260923 23:09:19.520629 23697 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37663
I20260923 23:09:19.520742 23697 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:19.520967 23697 heartbeater.cc:511] Master 127.5.74.62:37663 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
I20260923 23:09:19.521598 23505 ts_manager.cc:194] Registered new tserver with Master: 35659ef7d3ac4d219a536e770d1d5b71 (127.5.74.1:35899)
I20260923 23:09:19.522212 23505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:59431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:09:19.543367 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37663
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:19.626492 23702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:19.626747 23702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:19.626789 23702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:19.628026 23702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:19.628110 23702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:19.628159 23702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:09:19.630164 23702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:19.630510 23702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:19.630791 23702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:19.637023 23708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:19.637015 23710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:19.637015 23707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:19.637292 23702 server_base.cc:1034] running on GCE node
I20260923 23:09:19.637615 23702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:19.637893 23702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:19.639125 23702 hybrid_clock.cc:648] HybridClock initialized: now 1790204959639097 us; error 44 us; skew 500 ppm
Sep 23 23:09:19 dist-test-slave-j4c5 krb5kdc[23461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204959, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:19.641966 23702 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:09:19.642450 23702 webserver.cc:460] Webserver started at http://127.5.74.2:46857/ using document root <none> and password file <none>
I20260923 23:09:19.642714 23702 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:19.642791 23702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:19.642987 23702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:19.644129 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "636d4c1cc1af4505a8c6877abf5a5fbe"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "9b9010616ae751cb4e0a950611a1fdfc"
server_key_iv: "9479b4a5cb1a28e8cf911a801eaa88be"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.644582 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "636d4c1cc1af4505a8c6877abf5a5fbe"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "9b9010616ae751cb4e0a950611a1fdfc"
server_key_iv: "9479b4a5cb1a28e8cf911a801eaa88be"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.646054 23702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:09:19.646927 23717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.647244 23702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:19.647357 23702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "636d4c1cc1af4505a8c6877abf5a5fbe"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "9b9010616ae751cb4e0a950611a1fdfc"
server_key_iv: "9479b4a5cb1a28e8cf911a801eaa88be"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.647439 23702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:19.654659 23702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:19.656806 23702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:19.656972 23702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:19.657207 23702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:19.657528 23702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:19.657584 23702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.657626 23702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:19.657655 23702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.664042 23702 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:37543
I20260923 23:09:19.664090 23830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:37543 every 8 connection(s)
I20260923 23:09:19.664472 23702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:09:19.668699 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 23702
I20260923 23:09:19.668852 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:09:19.669096 5416 external_mini_cluster.cc:1428] Setting key b1ba3a4b40cd7be16420bf2c3b8bd7d6
Sep 23 23:09:19 dist-test-slave-j4c5 krb5kdc[23461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204959, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:19.676630 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.665437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:56075 (local address 127.5.74.62:37663)
0923 23:09:19.665485 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:19.665492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:19.665588 (+ 96us) server_negotiation.cc:407] Connection header received
0923 23:09:19.666166 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:19.666168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:19.666181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:19.666237 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:19.667790 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.668491 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.669984 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.670304 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.670419 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.672910 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:19.672912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:19.672912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:19.672927 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:19.674347 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.674812 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.674813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.674814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.674949 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.675308 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.675310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.675310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.675495 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:19.675576 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:19.676383 (+ 807us) server_negotiation.cc:299] Negotiation successful
0923 23:09:19.676532 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:09:19.676806 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.665280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37663 (local address 127.5.74.2:56075)
0923 23:09:19.665451 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:19.665495 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:19.666049 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:19.666318 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:19.666323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:19.667279 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:19.667639 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.667654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.668622 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.668623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.669854 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.669857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.670467 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.670468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.670577 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.671193 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:19.671203 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:19.672799 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:19.674476 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:19.674485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:19.674487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:19.674731 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:19.675096 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:19.675097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:19.675098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:19.675234 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:19.675577 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:19.675585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:19.676101 (+ 516us) client_negotiation.cc:769] Sending connection context
0923 23:09:19.676282 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 23:09:19.676548 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
I20260923 23:09:19.677325 23831 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37663
I20260923 23:09:19.677457 23831 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:19.677685 23831 heartbeater.cc:511] Master 127.5.74.62:37663 requested a full tablet report, sending...
I20260923 23:09:19.678210 23505 ts_manager.cc:194] Registered new tserver with Master: 636d4c1cc1af4505a8c6877abf5a5fbe (127.5.74.2:37543)
I20260923 23:09:19.678746 23505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:56075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:09:19.700541 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37663
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:19.782981 23836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:19.783205 23836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:19.783229 23836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:19.784461 23836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:19.784518 23836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:19.784545 23836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:09:19.786168 23836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:19.786450 23836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:19.786703 23836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:19.792152 23841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:19.792207 23842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:19.792313 23844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:19.792630 23836 server_base.cc:1034] running on GCE node
I20260923 23:09:19.792877 23836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:19.793115 23836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:19.794267 23836 hybrid_clock.cc:648] HybridClock initialized: now 1790204959794242 us; error 41 us; skew 500 ppm
Sep 23 23:09:19 dist-test-slave-j4c5 krb5kdc[23461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204959, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:19.797066 23836 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:09:19.797611 23836 webserver.cc:460] Webserver started at http://127.5.74.3:46303/ using document root <none> and password file <none>
I20260923 23:09:19.797868 23836 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:19.797940 23836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:19.798094 23836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:19.799202 23836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "ab42e3d531b44f5ebe18452d5d1abb6a"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "bc7f6054515dff412d0595075b877fc4"
server_key_iv: "6882ada5100ae03470d95f75fe910d2d"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.799598 23836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "ab42e3d531b44f5ebe18452d5d1abb6a"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "bc7f6054515dff412d0595075b877fc4"
server_key_iv: "6882ada5100ae03470d95f75fe910d2d"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.801121 23836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:19.802115 23851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.802305 23836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:19.802471 23836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "ab42e3d531b44f5ebe18452d5d1abb6a"
format_stamp: "Formatted at 2026-09-23 23:09:19 on dist-test-slave-j4c5"
server_key: "bc7f6054515dff412d0595075b877fc4"
server_key_iv: "6882ada5100ae03470d95f75fe910d2d"
server_key_version: "encryptionkey@0"
I20260923 23:09:19.802560 23836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:19.831704 23836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:19.834617 23836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:19.834818 23836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:19.835088 23836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:19.835434 23836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:19.835492 23836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.835536 23836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:19.835564 23836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:19.841965 23836 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:35479
I20260923 23:09:19.842003 23964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:35479 every 8 connection(s)
I20260923 23:09:19.842350 23836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:09:19.846521 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 23836
I20260923 23:09:19.846626 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:09:19.846823 5416 external_mini_cluster.cc:1428] Setting key 96554a7e7b77d56b072fbf2d71ad55ee
Sep 23 23:09:19 dist-test-slave-j4c5 krb5kdc[23461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204959, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:19.852617 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.843100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:50725 (local address 127.5.74.62:37663)
0923 23:09:19.843174 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:19.843176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:19.843497 (+ 321us) server_negotiation.cc:407] Connection header received
0923 23:09:19.844023 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:19.844025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:19.844035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:19.844093 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:19.845106 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.845647 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.846980 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.847233 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.847341 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.849665 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:19.849667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:19.849667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:19.849675 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:19.851084 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.851542 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.851544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.851544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.851603 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.851863 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.851865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.851865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.851991 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:19.852070 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:19.852437 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 23:09:19.852520 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 23:09:19.852823 23967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.843103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:37663 (local address 127.5.74.3:50725)
0923 23:09:19.843266 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:19.843302 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:19.843906 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:19.844181 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:19.844186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:19.844823 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:19.845016 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.845020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.845754 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.845755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.846834 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.846837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.847350 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.847351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.847506 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.848074 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:19.848083 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:19.849552 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:19.851191 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:19.851199 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:19.851202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:19.851443 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:19.851673 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:19.851674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:19.851674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:19.851794 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:19.852060 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:19.852067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:19.852292 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 23:09:19.852406 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 23:09:19.852578 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260923 23:09:19.853319 23965 heartbeater.cc:347] Connected to a master server at 127.5.74.62:37663
I20260923 23:09:19.853472 23965 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:19.853668 23965 heartbeater.cc:511] Master 127.5.74.62:37663 requested a full tablet report, sending...
I20260923 23:09:19.854045 23505 ts_manager.cc:194] Registered new tserver with Master: ab42e3d531b44f5ebe18452d5d1abb6a (127.5.74.3:35479)
I20260923 23:09:19.854450 23505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:50725
I20260923 23:09:19.858887 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:09:19.867601 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.860278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.5.74.62:37663)
0923 23:09:19.860449 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:19.860451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:19.860461 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:09:19.860515 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:19.860515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:19.860524 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:19.860576 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:19.861248 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.861837 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.862658 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.862869 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.862936 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.864553 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:19.864555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:19.864555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:19.864563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:19.866148 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.866613 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.866614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.866614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.866673 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:19.866960 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:19.866961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:19.866961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:19.867131 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:19.867253 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:19.867404 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 23:09:19.867461 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260923 23:09:19.869776 23505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:09:19.871465 23505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:09:19.878979 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.874455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:37543 (local address 127.0.0.1:45044)
0923 23:09:19.874861 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:19.874910 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:19.874981 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:19.875298 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:19.875302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:19.875304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:19.875429 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.875432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.876552 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.876552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.877808 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.877811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.878610 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.878611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.878697 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.878711 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:09:19.878816 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 23:09:19.878881 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260923 23:09:19.878991 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.874667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.5.74.2:37543)
0923 23:09:19.875136 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:19.875140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:19.875149 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:09:19.875179 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:19.875181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:19.875193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:19.875277 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:19.875529 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.876455 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.877899 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.878521 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.878612 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.878678 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:19.878851 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 23:09:19.878899 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":88,"threads_started":1}
I20260923 23:09:19.879199 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.874681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35899 (local address 127.0.0.1:44254)
0923 23:09:19.875009 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:19.875017 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:19.875059 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:19.875642 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:19.875642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:19.875644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:19.875713 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.875714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.877020 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.877021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.878060 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.878062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.879000 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.879000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.879057 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.879060 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:19.879102 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 23:09:19.879134 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260923 23:09:19.879400 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.874887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.5.74.1:35899)
0923 23:09:19.875418 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:19.875422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:19.875431 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:09:19.875473 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:19.875475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:19.875491 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:19.875563 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:19.875805 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.876919 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.878149 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.878841 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.878918 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.878970 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:19.879271 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 23:09:19.879321 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":99,"threads_started":1}
I20260923 23:09:19.879462 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.874836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.5.74.3:35479)
0923 23:09:19.875162 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:19.875166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:19.875194 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:09:19.875233 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:19.875234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:19.875250 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:19.875324 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:19.875553 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.876716 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.878007 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.878745 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.878819 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.878876 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:19.879321 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 23:09:19.879365 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":86,"threads_started":1}
I20260923 23:09:19.879549 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:19.874455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:35479 (local address 127.0.0.1:47570)
0923 23:09:19.874865 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:19.874909 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:19.874980 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:19.875416 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:19.875416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:19.875418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:19.875475 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.875476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.876826 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.876827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.877923 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:19.877925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:19.878914 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:19.878915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:19.878984 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:19.878988 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:09:19.879036 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 23:09:19.879084 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260923 23:09:19.880402 23765 tablet_service.cc:1467] Processing CreateTablet for tablet f6d30a3bae9b4578a1bd0c6532bc178b (DEFAULT_TABLE table=test-table [id=6faf516b47164e55beb6abf04156b5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:19.880417 23899 tablet_service.cc:1467] Processing CreateTablet for tablet f6d30a3bae9b4578a1bd0c6532bc178b (DEFAULT_TABLE table=test-table [id=6faf516b47164e55beb6abf04156b5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:19.880559 23631 tablet_service.cc:1467] Processing CreateTablet for tablet f6d30a3bae9b4578a1bd0c6532bc178b (DEFAULT_TABLE table=test-table [id=6faf516b47164e55beb6abf04156b5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:19.880695 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d30a3bae9b4578a1bd0c6532bc178b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:19.880740 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d30a3bae9b4578a1bd0c6532bc178b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:19.880836 23631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d30a3bae9b4578a1bd0c6532bc178b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:19.883680 23982 tablet_bootstrap.cc:492] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: Bootstrap starting.
I20260923 23:09:19.884255 23983 tablet_bootstrap.cc:492] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71: Bootstrap starting.
I20260923 23:09:19.884475 23982 tablet_bootstrap.cc:654] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:19.884732 23984 tablet_bootstrap.cc:492] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: Bootstrap starting.
I20260923 23:09:19.885066 23982 log.cc:824] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:19.885499 23983 tablet_bootstrap.cc:654] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:19.885900 23982 tablet_bootstrap.cc:492] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: No bootstrap required, opened a new log
I20260923 23:09:19.885962 23982 ts_tablet_manager.cc:1397] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:09:19.886068 23983 log.cc:824] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:19.886523 23984 tablet_bootstrap.cc:654] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:19.886977 23984 log.cc:824] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:19.887805 23984 tablet_bootstrap.cc:492] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: No bootstrap required, opened a new log
I20260923 23:09:19.887817 23983 tablet_bootstrap.cc:492] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71: No bootstrap required, opened a new log
I20260923 23:09:19.887885 23984 ts_tablet_manager.cc:1397] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:09:19.887895 23983 ts_tablet_manager.cc:1397] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 23:09:19.887814 23982 raft_consensus.cc:348] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } }
I20260923 23:09:19.888022 23982 raft_consensus.cc:374] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:19.888070 23982 raft_consensus.cc:729] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636d4c1cc1af4505a8c6877abf5a5fbe, State: Initialized, Role: FOLLOWER
I20260923 23:09:19.888207 23982 consensus_queue.cc:260] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } }
I20260923 23:09:19.888507 23982 ts_tablet_manager.cc:1428] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260923 23:09:19.888644 23831 heartbeater.cc:503] Master 127.5.74.62:37663 was elected leader, sending a full tablet report...
I20260923 23:09:19.889880 23984 raft_consensus.cc:348] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } }
I20260923 23:09:19.889899 23983 raft_consensus.cc:348] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } }
I20260923 23:09:19.890064 23984 raft_consensus.cc:374] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:19.890062 23983 raft_consensus.cc:374] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:19.890100 23983 raft_consensus.cc:729] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35659ef7d3ac4d219a536e770d1d5b71, State: Initialized, Role: FOLLOWER
I20260923 23:09:19.890100 23984 raft_consensus.cc:729] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab42e3d531b44f5ebe18452d5d1abb6a, State: Initialized, Role: FOLLOWER
I20260923 23:09:19.890211 23984 consensus_queue.cc:260] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } }
I20260923 23:09:19.890211 23983 consensus_queue.cc:260] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } }
I20260923 23:09:19.890497 23983 ts_tablet_manager.cc:1428] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:09:19.890497 23984 ts_tablet_manager.cc:1428] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:09:19.890568 23965 heartbeater.cc:503] Master 127.5.74.62:37663 was elected leader, sending a full tablet report...
I20260923 23:09:19.890808 23697 heartbeater.cc:503] Master 127.5.74.62:37663 was elected leader, sending a full tablet report...
W20260923 23:09:19.915127 23832 tablet.cc:2369] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:09:20.010254 23698 tablet.cc:2369] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:09:20.092962 23966 tablet.cc:2369] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:09:20.139120 23989 raft_consensus.cc:482] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:20.139268 23989 raft_consensus.cc:504] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } }
I20260923 23:09:20.139737 23989 leader_election.cc:288] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 636d4c1cc1af4505a8c6877abf5a5fbe (127.5.74.2:37543), ab42e3d531b44f5ebe18452d5d1abb6a (127.5.74.3:35479)
I20260923 23:09:20.144248 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.140044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:39947 (local address 127.5.74.2:37543)
0923 23:09:20.140115 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.140118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.140225 (+ 107us) server_negotiation.cc:407] Connection header received
0923 23:09:20.140349 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.140350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.140364 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.140419 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:20.140669 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.141389 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.143165 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.143791 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.143867 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.143880 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.144087 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.144133 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:09:20.144248 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.140247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:57355 (local address 127.5.74.3:35479)
0923 23:09:20.140297 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.140299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.140306 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:09:20.140336 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.140336 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.140348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.140411 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:20.140620 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.141409 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.143165 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.143773 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.143867 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.143880 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.144087 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.144131 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:09:20.144306 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.139939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:37543 (local address 127.5.74.1:39947)
0923 23:09:20.140123 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.140136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.140243 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.140440 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.140442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.140445 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:20.140593 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.140595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.141528 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.141529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.143023 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.143026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.143896 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.143897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.143988 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.143996 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.144062 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.144127 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":196,"thread_start_us":114,"threads_started":1}
I20260923 23:09:20.144304 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.140030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:35479 (local address 127.5.74.1:57355)
0923 23:09:20.140185 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.140195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.140254 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.140442 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.140443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.140445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:20.140529 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.140531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.141516 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.141518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.143021 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.143026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.143894 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.143896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.143988 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.143995 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.144062 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.144127 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":133,"thread_start_us":81,"threads_started":1}
I20260923 23:09:20.144752 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d30a3bae9b4578a1bd0c6532bc178b" candidate_uuid: "35659ef7d3ac4d219a536e770d1d5b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" is_pre_election: true
I20260923 23:09:20.144860 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d30a3bae9b4578a1bd0c6532bc178b" candidate_uuid: "35659ef7d3ac4d219a536e770d1d5b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" is_pre_election: true
I20260923 23:09:20.144943 23785 raft_consensus.cc:2454] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35659ef7d3ac4d219a536e770d1d5b71 in term 0.
I20260923 23:09:20.145046 23919 raft_consensus.cc:2454] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35659ef7d3ac4d219a536e770d1d5b71 in term 0.
I20260923 23:09:20.145205 23587 leader_election.cc:302] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35659ef7d3ac4d219a536e770d1d5b71, 636d4c1cc1af4505a8c6877abf5a5fbe; no voters:
I20260923 23:09:20.145358 23989 raft_consensus.cc:2781] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:09:20.145418 23989 raft_consensus.cc:482] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:09:20.145442 23989 raft_consensus.cc:3037] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:20.146224 23989 raft_consensus.cc:504] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } }
I20260923 23:09:20.146463 23989 leader_election.cc:288] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [CANDIDATE]: Term 1 election: Requested vote from peers 636d4c1cc1af4505a8c6877abf5a5fbe (127.5.74.2:37543), ab42e3d531b44f5ebe18452d5d1abb6a (127.5.74.3:35479)
I20260923 23:09:20.146584 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d30a3bae9b4578a1bd0c6532bc178b" candidate_uuid: "35659ef7d3ac4d219a536e770d1d5b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a"
I20260923 23:09:20.146584 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d30a3bae9b4578a1bd0c6532bc178b" candidate_uuid: "35659ef7d3ac4d219a536e770d1d5b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe"
I20260923 23:09:20.146678 23919 raft_consensus.cc:3037] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:20.146678 23785 raft_consensus.cc:3037] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:20.147625 23785 raft_consensus.cc:2454] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35659ef7d3ac4d219a536e770d1d5b71 in term 1.
I20260923 23:09:20.147636 23919 raft_consensus.cc:2454] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35659ef7d3ac4d219a536e770d1d5b71 in term 1.
I20260923 23:09:20.147850 23585 leader_election.cc:302] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35659ef7d3ac4d219a536e770d1d5b71, ab42e3d531b44f5ebe18452d5d1abb6a; no voters:
I20260923 23:09:20.147979 23989 raft_consensus.cc:2781] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:20.148201 23989 raft_consensus.cc:686] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [term 1 LEADER]: Becoming Leader. State: Replica: 35659ef7d3ac4d219a536e770d1d5b71, State: Running, Role: LEADER
I20260923 23:09:20.148311 23989 consensus_queue.cc:237] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } }
I20260923 23:09:20.149235 23506 catalog_manager.cc:5495] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35659ef7d3ac4d219a536e770d1d5b71 (127.5.74.1). New cstate: current_term: 1 leader_uuid: "35659ef7d3ac4d219a536e770d1d5b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35659ef7d3ac4d219a536e770d1d5b71" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 35899 } health_report { overall_health: HEALTHY } } }
I20260923 23:09:20.164242 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.161780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.5.74.1:35899)
0923 23:09:20.161849 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.161852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.161862 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:09:20.161910 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.161911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.161931 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.162012 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:20.162156 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.162668 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.163421 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.163616 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.163691 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.163863 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:20.163957 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:20.164070 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.164105 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.164113 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 23:09:20.166520 23919 raft_consensus.cc:1261] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [term 1 FOLLOWER]: Refusing update from remote peer 35659ef7d3ac4d219a536e770d1d5b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:20.166687 23785 raft_consensus.cc:1261] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [term 1 FOLLOWER]: Refusing update from remote peer 35659ef7d3ac4d219a536e770d1d5b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:20.166953 23989 consensus_queue.cc:1035] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab42e3d531b44f5ebe18452d5d1abb6a" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 35479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:20.167106 23993 consensus_queue.cc:1035] T f6d30a3bae9b4578a1bd0c6532bc178b P 35659ef7d3ac4d219a536e770d1d5b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "636d4c1cc1af4505a8c6877abf5a5fbe" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:20.170409 23998 mvcc.cc:205] Tried to move back new op lower bound from 7332679516838346752 to 7332679516768247808. Current Snapshot: MvccSnapshot[applied={T|T < 7332679516838346752}]
I20260923 23:09:20.170262 23997 mvcc.cc:205] Tried to move back new op lower bound from 7332679516838346752 to 7332679516768247808. Current Snapshot: MvccSnapshot[applied={T|T < 7332679516838346752}]
I20260923 23:09:20.171959 23506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:20.172075 23506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:20.172962 23506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e89afc85b1ba4f77848d55b5518f6824: Sending DeleteTablet for 3 replicas of tablet f6d30a3bae9b4578a1bd0c6532bc178b
I20260923 23:09:20.173448 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d30a3bae9b4578a1bd0c6532bc178b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:20 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:47570
I20260923 23:09:20.173568 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d30a3bae9b4578a1bd0c6532bc178b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:20 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:45044
I20260923 23:09:20.173827 24007 tablet_replica.cc:330] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: stopping tablet replica
I20260923 23:09:20.173828 23631 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d30a3bae9b4578a1bd0c6532bc178b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:20 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:44254
I20260923 23:09:20.173869 24006 tablet_replica.cc:330] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: stopping tablet replica
I20260923 23:09:20.173969 24006 raft_consensus.cc:2229] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:09:20.173983 24007 raft_consensus.cc:2229] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:09:20.174069 24006 raft_consensus.cc:2258] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:20.174096 24007 raft_consensus.cc:2258] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:20.174171 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 23568
I20260923 23:09:20.175133 24007 ts_tablet_manager.cc:1905] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:09:20.175967 24006 ts_tablet_manager.cc:1905] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:09:20.177466 24007 ts_tablet_manager.cc:1918] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:20.177601 24007 log.cc:1196] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/f6d30a3bae9b4578a1bd0c6532bc178b
I20260923 23:09:20.177773 24006 ts_tablet_manager.cc:1918] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:20.177877 24006 log.cc:1196] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/f6d30a3bae9b4578a1bd0c6532bc178b
I20260923 23:09:20.177877 24007 ts_tablet_manager.cc:1939] T f6d30a3bae9b4578a1bd0c6532bc178b P 636d4c1cc1af4505a8c6877abf5a5fbe: Deleting consensus metadata
I20260923 23:09:20.178182 24006 ts_tablet_manager.cc:1939] T f6d30a3bae9b4578a1bd0c6532bc178b P ab42e3d531b44f5ebe18452d5d1abb6a: Deleting consensus metadata
I20260923 23:09:20.178463 23493 catalog_manager.cc:4841] TS 636d4c1cc1af4505a8c6877abf5a5fbe (127.5.74.2:37543): tablet f6d30a3bae9b4578a1bd0c6532bc178b (table test-table [id=6faf516b47164e55beb6abf04156b5e3]) successfully deleted
I20260923 23:09:20.178642 23491 catalog_manager.cc:4841] TS ab42e3d531b44f5ebe18452d5d1abb6a (127.5.74.3:35479): tablet f6d30a3bae9b4578a1bd0c6532bc178b (table test-table [id=6faf516b47164e55beb6abf04156b5e3]) successfully deleted
W20260923 23:09:20.181111 23491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:35899 (error 108)
I20260923 23:09:20.181209 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 23702
I20260923 23:09:20.186662 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 23836
I20260923 23:09:20.186981 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.186729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:35899 (local address 127.0.0.1:44266)
0923 23:09:20.186869 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.186928 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:35899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 23:09:20.187829 23491 catalog_manager.cc:4584] TS 35659ef7d3ac4d219a536e770d1d5b71 (127.5.74.1:35899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6d30a3bae9b4578a1bd0c6532bc178b: Network error: Client connection negotiation failed: client connection to 127.5.74.1:35899: connect: Connection refused (error 111)
I20260923 23:09:20.191938 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 23474
2026-09-23T23:09:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2191 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T23:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:20Z Disabled control of system clock
I20260923 23:09:20.221086 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:39727
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:09:20.302913 24013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:20.303174 24013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:20.303197 24013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:20.304451 24013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:20.304497 24013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:20.304510 24013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:20.304524 24013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:20.304536 24013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:20.305909 24013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:39727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:20.306137 24013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:20.306406 24013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:09:20.312011 24013 server_base.cc:1034] running on GCE node
W20260923 23:09:20.311973 24019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:20.311960 24021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:20.311957 24018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:20.312683 24013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:20.313025 24013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:20.314186 24013 hybrid_clock.cc:648] HybridClock initialized: now 1790204960314166 us; error 45 us; skew 500 ppm
I20260923 23:09:20.315672 24013 webserver.cc:460] Webserver started at http://127.5.74.62:37821/ using document root <none> and password file <none>
I20260923 23:09:20.315963 24013 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:20.316016 24013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:20.316188 24013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:20.317299 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "51e42530ba814ab3ab966763abeac8c9"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "78eccfb4b46297d96569df8ce8b2cc50"
server_key_iv: "f7f1817f451a55563cdd5e277a6a0823"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.317849 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "51e42530ba814ab3ab966763abeac8c9"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "78eccfb4b46297d96569df8ce8b2cc50"
server_key_iv: "f7f1817f451a55563cdd5e277a6a0823"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.319458 24013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:20.320495 24027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.320711 24013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:20.320822 24013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "51e42530ba814ab3ab966763abeac8c9"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "78eccfb4b46297d96569df8ce8b2cc50"
server_key_iv: "f7f1817f451a55563cdd5e277a6a0823"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.320920 24013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:20.334304 24013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:20.334693 24013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:20.334859 24013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:20.339610 24013 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:39727
I20260923 23:09:20.339643 24079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:39727 every 8 connection(s)
I20260923 23:09:20.340044 24013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:20.341146 24080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:20.343473 24080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9: Bootstrap starting.
I20260923 23:09:20.344054 24080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:20.344420 24080 log.cc:824] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:20.345176 24080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9: No bootstrap required, opened a new log
I20260923 23:09:20.345958 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 24013
I20260923 23:09:20.346104 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:20.346402 5416 external_mini_cluster.cc:1428] Setting key 52c6e59e9e48bdf34f43f5a6c298e67a
I20260923 23:09:20.347271 24080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e42530ba814ab3ab966763abeac8c9" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39727 } }
I20260923 23:09:20.347451 24080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:20.347497 24080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e42530ba814ab3ab966763abeac8c9, State: Initialized, Role: FOLLOWER
I20260923 23:09:20.347661 24080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e42530ba814ab3ab966763abeac8c9" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39727 } }
I20260923 23:09:20.347756 24080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:20.347791 24080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:20.347855 24080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:20.348711 24080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e42530ba814ab3ab966763abeac8c9" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39727 } }
I20260923 23:09:20.348840 24080 leader_election.cc:302] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51e42530ba814ab3ab966763abeac8c9; no voters:
I20260923 23:09:20.349009 24080 leader_election.cc:288] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:20.349105 24085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:20.349293 24085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [term 1 LEADER]: Becoming Leader. State: Replica: 51e42530ba814ab3ab966763abeac8c9, State: Running, Role: LEADER
I20260923 23:09:20.349355 24080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:20.349426 24085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e42530ba814ab3ab966763abeac8c9" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39727 } }
I20260923 23:09:20.349987 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e42530ba814ab3ab966763abeac8c9. Latest consensus state: current_term: 1 leader_uuid: "51e42530ba814ab3ab966763abeac8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e42530ba814ab3ab966763abeac8c9" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39727 } } }
I20260923 23:09:20.350068 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:20.350395 24086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51e42530ba814ab3ab966763abeac8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e42530ba814ab3ab966763abeac8c9" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39727 } } }
I20260923 23:09:20.350466 24086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:20.351127 24093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:20.351621 24093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:20.353572 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.347043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.5.74.62:39727)
0923 23:09:20.347280 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.347290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.347318 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:09:20.347421 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.347427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.347505 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.347672 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:20.347924 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.349828 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.351283 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.351783 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.351934 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.352745 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:20.352749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:20.352749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:20.352978 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:20.353046 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:20.353062 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:20.353149 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.353233 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.353326 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":119,"threads_started":1}
I20260923 23:09:20.358932 24093 catalog_manager.cc:1347] Generated new cluster ID: 2b8c52c315144aff857d98d9ef2aeea9
I20260923 23:09:20.359045 24093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:20.383697 24093 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:20.384581 24093 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:20.388424 24093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9: Generated new TSK 0
I20260923 23:09:20.388684 24093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:20.394830 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39727
--builtin_ntp_servers=127.5.74.20:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:09:20.498366 24104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:20.498674 24104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:20.498713 24104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:20.499975 24104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:20.500053 24104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:20.500099 24104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:09:20.501678 24104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:20.502007 24104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:20.502357 24104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:20.507650 24110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:20.507671 24109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:20.507834 24104 server_base.cc:1034] running on GCE node
W20260923 23:09:20.507687 24112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:20.508121 24104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:20.508428 24104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:20.509596 24104 hybrid_clock.cc:648] HybridClock initialized: now 1790204960509566 us; error 44 us; skew 500 ppm
I20260923 23:09:20.511121 24104 webserver.cc:460] Webserver started at http://127.5.74.1:32965/ using document root <none> and password file <none>
I20260923 23:09:20.511405 24104 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:20.511480 24104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:20.511631 24104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:20.512866 24104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "7a9e7c7f5a354988807f499366471333"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "1eb527b92f3204c2212986bbf7542f20"
server_key_iv: "2ff8c997b6f93435f03b334f847115d7"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.513253 24104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "7a9e7c7f5a354988807f499366471333"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "1eb527b92f3204c2212986bbf7542f20"
server_key_iv: "2ff8c997b6f93435f03b334f847115d7"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.514688 24104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:09:20.515653 24118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.516019 24104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:20.516191 24104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "7a9e7c7f5a354988807f499366471333"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "1eb527b92f3204c2212986bbf7542f20"
server_key_iv: "2ff8c997b6f93435f03b334f847115d7"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.516289 24104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:20.528784 24104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:20.529171 24104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:20.529385 24104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:20.529670 24104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:20.530048 24104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:20.530112 24104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.530158 24104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:20.530208 24104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.537102 24104 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:42587
I20260923 23:09:20.537156 24231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:42587 every 8 connection(s)
I20260923 23:09:20.537544 24104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:09:20.541388 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 24104
I20260923 23:09:20.541512 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:09:20.541762 5416 external_mini_cluster.cc:1428] Setting key 349f0d9305182ee80b03ac91dd7e050a
I20260923 23:09:20.543062 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39727
--builtin_ntp_servers=127.5.74.20:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:09:20.544288 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.538451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:57309 (local address 127.5.74.62:39727)
0923 23:09:20.538542 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.538544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.538778 (+ 234us) server_negotiation.cc:407] Connection header received
0923 23:09:20.539255 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.539256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.539270 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.539340 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:20.540094 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.540682 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.542215 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.542627 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.542723 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.543847 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:20.543849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:20.543849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:20.543855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:20.543873 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:20.543879 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:20.543941 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.544111 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.544210 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 23:09:20.544637 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.538404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39727 (local address 127.5.74.1:57309)
0923 23:09:20.538621 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.538664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.539166 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.539438 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.539442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.539451 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:20.539988 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.539993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.540790 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.540791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.542062 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.542066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.542776 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.542778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.542985 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.543649 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:09:20.543649 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:20.543695 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:09:20.543730 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:20.543990 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:20.543997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:20.544001 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.544082 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.544263 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260923 23:09:20.545238 24232 heartbeater.cc:347] Connected to a master server at 127.5.74.62:39727
I20260923 23:09:20.545338 24232 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:20.545564 24232 heartbeater.cc:511] Master 127.5.74.62:39727 requested a full tablet report, sending...
I20260923 23:09:20.546212 24044 ts_manager.cc:194] Registered new tserver with Master: 7a9e7c7f5a354988807f499366471333 (127.5.74.1:42587)
I20260923 23:09:20.546871 24044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.1:57309
W20260923 23:09:20.628190 24235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:20.628474 24235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:20.628515 24235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:20.629817 24235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:20.629907 24235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:20.629951 24235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:09:20.631528 24235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:20.631868 24235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:20.632239 24235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:20.637774 24243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:20.637826 24241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:20.637840 24240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:20.638089 24235 server_base.cc:1034] running on GCE node
I20260923 23:09:20.638322 24235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:20.638623 24235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:20.639829 24235 hybrid_clock.cc:648] HybridClock initialized: now 1790204960639801 us; error 66 us; skew 500 ppm
I20260923 23:09:20.641677 24235 webserver.cc:460] Webserver started at http://127.5.74.2:41555/ using document root <none> and password file <none>
I20260923 23:09:20.641985 24235 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:20.642072 24235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:20.642233 24235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:20.643559 24235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "c3bb25d996424a219219b152722d456b"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "de97cdf4e8e8abe2c9403fed5fdac26f"
server_key_iv: "ad60097600a7ba3341881de419238afd"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.643965 24235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "c3bb25d996424a219219b152722d456b"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "de97cdf4e8e8abe2c9403fed5fdac26f"
server_key_iv: "ad60097600a7ba3341881de419238afd"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.645596 24235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:20.646479 24249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.646727 24235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:20.646842 24235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "c3bb25d996424a219219b152722d456b"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "de97cdf4e8e8abe2c9403fed5fdac26f"
server_key_iv: "ad60097600a7ba3341881de419238afd"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.646939 24235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:20.662137 24235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:20.662526 24235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:20.662705 24235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:20.662967 24235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:20.663321 24235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:20.663388 24235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.663440 24235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:20.663466 24235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.670100 24235 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:34311
I20260923 23:09:20.670141 24362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:34311 every 8 connection(s)
I20260923 23:09:20.670522 24235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:09:20.676527 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.671389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:35783 (local address 127.5.74.62:39727)
0923 23:09:20.671463 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.671466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.671659 (+ 193us) server_negotiation.cc:407] Connection header received
0923 23:09:20.672229 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.672230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.672241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.672301 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:20.673030 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.673632 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.674878 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.675094 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.675164 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.676175 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:20.676176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:20.676177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:20.676182 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:20.676197 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:20.676201 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:20.676250 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.676428 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.676457 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:09:20.676734 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.671378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39727 (local address 127.5.74.2:35783)
0923 23:09:20.671555 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.671588 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.672068 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.672387 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.672392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.672403 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:20.672920 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.672924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.673761 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.673763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.674783 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.674786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.675202 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.675204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.675351 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.675994 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:09:20.675995 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:20.676025 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:09:20.676067 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:20.676284 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:20.676290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:20.676300 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.676411 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.676504 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260923 23:09:20.677249 24363 heartbeater.cc:347] Connected to a master server at 127.5.74.62:39727
I20260923 23:09:20.677350 24363 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:20.677507 24363 heartbeater.cc:511] Master 127.5.74.62:39727 requested a full tablet report, sending...
I20260923 23:09:20.677891 24044 ts_manager.cc:194] Registered new tserver with Master: c3bb25d996424a219219b152722d456b (127.5.74.2:34311)
I20260923 23:09:20.678234 24044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.2:35783
I20260923 23:09:20.679596 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 24235
I20260923 23:09:20.679677 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:09:20.679834 5416 external_mini_cluster.cc:1428] Setting key f4bde7dec2c281c8e36a15c775f0e845
I20260923 23:09:20.680778 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39727
--builtin_ntp_servers=127.5.74.20:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:09:20.765148 24366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:20.765414 24366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:20.765457 24366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:20.766652 24366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:20.766728 24366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:20.766772 24366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:09:20.768299 24366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:20.768580 24366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:20.768903 24366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:20.774286 24371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:20.774269 24372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:20.774271 24374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:20.774596 24366 server_base.cc:1034] running on GCE node
I20260923 23:09:20.774888 24366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:20.775161 24366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:20.776324 24366 hybrid_clock.cc:648] HybridClock initialized: now 1790204960776310 us; error 33 us; skew 500 ppm
I20260923 23:09:20.777622 24366 webserver.cc:460] Webserver started at http://127.5.74.3:42861/ using document root <none> and password file <none>
I20260923 23:09:20.777959 24366 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:20.778038 24366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:20.778196 24366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:20.779292 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "ddd459155b684826be9e4c08e75e1a27"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "4f4c8ba2e852d66b5ea2920854a5ce4b"
server_key_iv: "e5dd7f8cdd28fb36177b5d429b7ed0dd"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.779661 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "ddd459155b684826be9e4c08e75e1a27"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "4f4c8ba2e852d66b5ea2920854a5ce4b"
server_key_iv: "e5dd7f8cdd28fb36177b5d429b7ed0dd"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.781183 24366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:09:20.782121 24380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.782349 24366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:20.782450 24366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "ddd459155b684826be9e4c08e75e1a27"
format_stamp: "Formatted at 2026-09-23 23:09:20 on dist-test-slave-j4c5"
server_key: "4f4c8ba2e852d66b5ea2920854a5ce4b"
server_key_iv: "e5dd7f8cdd28fb36177b5d429b7ed0dd"
server_key_version: "encryptionkey@0"
I20260923 23:09:20.782538 24366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:20.794796 24366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:20.795147 24366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:20.795305 24366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:20.795552 24366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:20.795939 24366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:20.795996 24366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.796060 24366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:20.796090 24366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:20.801901 24366 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:39587
I20260923 23:09:20.801954 24493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:39587 every 8 connection(s)
I20260923 23:09:20.802338 24366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:09:20.806058 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 24366
I20260923 23:09:20.806193 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:09:20.806427 5416 external_mini_cluster.cc:1428] Setting key 6566a188c278fc417488b8227e8fe461
I20260923 23:09:20.808357 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.803160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:51445 (local address 127.5.74.62:39727)
0923 23:09:20.803239 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.803242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.803507 (+ 265us) server_negotiation.cc:407] Connection header received
0923 23:09:20.803978 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.803979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.803991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.804072 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:20.804877 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.805450 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.806696 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.807001 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.807087 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.808011 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:20.808012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:20.808013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:20.808018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:20.808033 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:20.808038 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:20.808116 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.808238 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.808273 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 23:09:20.808650 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.803129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:39727 (local address 127.5.74.3:51445)
0923 23:09:20.803343 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.803390 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.803882 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.804140 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.804143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.804196 (+ 53us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:20.804766 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.804771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.805568 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.805569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.806600 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.806602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.807100 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.807101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.807240 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.807821 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:09:20.807822 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:20.807858 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:09:20.807887 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:20.808116 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:20.808123 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:20.808126 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.808242 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.808390 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260923 23:09:20.809090 24494 heartbeater.cc:347] Connected to a master server at 127.5.74.62:39727
I20260923 23:09:20.809185 24494 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:20.809376 24494 heartbeater.cc:511] Master 127.5.74.62:39727 requested a full tablet report, sending...
I20260923 23:09:20.809868 24044 ts_manager.cc:194] Registered new tserver with Master: ddd459155b684826be9e4c08e75e1a27 (127.5.74.3:39587)
I20260923 23:09:20.810240 24044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.3:51445
I20260923 23:09:20.818698 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:09:20.823578 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.820274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.5.74.62:39727)
0923 23:09:20.820348 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.820351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.820534 (+ 183us) server_negotiation.cc:407] Connection header received
0923 23:09:20.820618 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.820619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.820631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.820702 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:20.820893 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.821392 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.822177 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.822374 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.822452 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.823231 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:20.823233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:20.823233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:20.823238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:20.823254 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:20.823258 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:20.823371 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.823462 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.823482 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:09:20.825502 24044 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:09:20.827049 24044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:09:20.833616 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.829529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:39587 (local address 127.0.0.1:52528)
0923 23:09:20.830172 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.830182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.830239 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.830513 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.830514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.830516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:20.830564 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.830566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.831414 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.831415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.832407 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.832410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.833301 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.833302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.833384 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.833397 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.833472 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.833523 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260923 23:09:20.833659 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.829657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.5.74.3:39587)
0923 23:09:20.829927 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.829931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.830304 (+ 373us) server_negotiation.cc:407] Connection header received
0923 23:09:20.830352 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.830355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.830370 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.830443 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:20.830624 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.831310 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.832506 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.833105 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.833174 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.833266 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.833517 (+ 251us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.833565 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260923 23:09:20.834200 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.829529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:34311 (local address 127.0.0.1:45582)
0923 23:09:20.829711 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.829750 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.829847 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.830096 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.830099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.830102 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:20.830223 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.830226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.831593 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.831594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.832900 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.832904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.833958 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.833959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.834017 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.834020 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.834074 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.834117 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"spinlock_wait_cycles":384,"thread_start_us":68,"threads_started":1}
I20260923 23:09:20.834254 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.829729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.5.74.2:34311)
0923 23:09:20.829896 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.829902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.829914 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:09:20.829958 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.829960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.829976 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.830090 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:20.830322 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.831485 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.833005 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.833768 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.833844 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.833926 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.834106 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.834148 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260923 23:09:20.834574 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.829759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:42587 (local address 127.0.0.1:33186)
0923 23:09:20.830196 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.830206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.830259 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.830537 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.830538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.830540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:20.830577 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.830579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.831863 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.831864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.833087 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.833089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.834108 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.834110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.834167 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.834171 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.834220 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.834261 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":82,"threads_started":1}
I20260923 23:09:20.834867 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.829971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.5.74.1:42587)
0923 23:09:20.830153 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.830158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.830325 (+ 167us) server_negotiation.cc:407] Connection header received
0923 23:09:20.830372 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.830374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.830389 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.830454 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:20.830656 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.831765 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.833170 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.833905 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.833980 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.834071 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.834745 (+ 674us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.834790 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
I20260923 23:09:20.834945 24297 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8802c13b2340dfb346300057326589 (DEFAULT_TABLE table=test-table [id=0e666d85b79d4f429b6122c8474f7ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:20.834949 24428 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8802c13b2340dfb346300057326589 (DEFAULT_TABLE table=test-table [id=0e666d85b79d4f429b6122c8474f7ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:20.835297 24428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8802c13b2340dfb346300057326589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:20.835306 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8802c13b2340dfb346300057326589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:20.835529 24166 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8802c13b2340dfb346300057326589 (DEFAULT_TABLE table=test-table [id=0e666d85b79d4f429b6122c8474f7ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:20.835820 24166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8802c13b2340dfb346300057326589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:20.838264 24508 tablet_bootstrap.cc:492] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: Bootstrap starting.
I20260923 23:09:20.838732 24509 tablet_bootstrap.cc:492] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b: Bootstrap starting.
I20260923 23:09:20.839128 24508 tablet_bootstrap.cc:654] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:20.839171 24510 tablet_bootstrap.cc:492] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333: Bootstrap starting.
I20260923 23:09:20.839377 24509 tablet_bootstrap.cc:654] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:20.839586 24508 log.cc:824] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:20.839684 24509 log.cc:824] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:20.839872 24510 tablet_bootstrap.cc:654] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:20.840243 24510 log.cc:824] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:20.840485 24508 tablet_bootstrap.cc:492] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: No bootstrap required, opened a new log
I20260923 23:09:20.840495 24509 tablet_bootstrap.cc:492] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b: No bootstrap required, opened a new log
I20260923 23:09:20.840559 24509 ts_tablet_manager.cc:1397] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:09:20.840559 24508 ts_tablet_manager.cc:1397] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:09:20.841019 24510 tablet_bootstrap.cc:492] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333: No bootstrap required, opened a new log
I20260923 23:09:20.841161 24510 ts_tablet_manager.cc:1397] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:09:20.842558 24508 raft_consensus.cc:348] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } }
I20260923 23:09:20.842725 24508 raft_consensus.cc:374] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:20.842769 24508 raft_consensus.cc:729] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd459155b684826be9e4c08e75e1a27, State: Initialized, Role: FOLLOWER
I20260923 23:09:20.842893 24508 consensus_queue.cc:260] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } }
I20260923 23:09:20.843205 24494 heartbeater.cc:503] Master 127.5.74.62:39727 was elected leader, sending a full tablet report...
I20260923 23:09:20.843269 24509 raft_consensus.cc:348] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } }
I20260923 23:09:20.843456 24509 raft_consensus.cc:374] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:20.843497 24509 raft_consensus.cc:729] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3bb25d996424a219219b152722d456b, State: Initialized, Role: FOLLOWER
I20260923 23:09:20.843611 24509 consensus_queue.cc:260] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } }
I20260923 23:09:20.843194 24508 ts_tablet_manager.cc:1428] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 23:09:20.843859 24509 ts_tablet_manager.cc:1428] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:09:20.843912 24363 heartbeater.cc:503] Master 127.5.74.62:39727 was elected leader, sending a full tablet report...
I20260923 23:09:20.843876 24510 raft_consensus.cc:348] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } }
I20260923 23:09:20.844012 24510 raft_consensus.cc:374] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:20.844043 24510 raft_consensus.cc:729] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9e7c7f5a354988807f499366471333, State: Initialized, Role: FOLLOWER
I20260923 23:09:20.844201 24510 consensus_queue.cc:260] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } }
I20260923 23:09:20.844480 24510 ts_tablet_manager.cc:1428] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:09:20.844518 24232 heartbeater.cc:503] Master 127.5.74.62:39727 was elected leader, sending a full tablet report...
I20260923 23:09:20.869511 24515 raft_consensus.cc:482] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:20.869643 24515 raft_consensus.cc:504] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } }
I20260923 23:09:20.870116 24515 leader_election.cc:288] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddd459155b684826be9e4c08e75e1a27 (127.5.74.3:39587), 7a9e7c7f5a354988807f499366471333 (127.5.74.1:42587)
I20260923 23:09:20.873114 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.870358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:42587 (local address 127.5.74.2:45971)
0923 23:09:20.870421 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.870437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.870502 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.870663 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.870664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.870666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:20.870742 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.870744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.871429 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.871430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.872274 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.872277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.872835 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.872837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.872911 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.872919 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.872994 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.873037 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 23:09:20.873113 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.870487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:45971 (local address 127.5.74.1:42587)
0923 23:09:20.870532 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.870534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.870544 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:09:20.870572 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.870573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.870584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.870660 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:20.870854 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.871340 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.872367 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.872737 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.872835 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.872847 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.873002 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.873045 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:09:20.873507 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.870397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:39587 (local address 127.5.74.2:46629)
0923 23:09:20.870604 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.870614 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:20.870699 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:20.870892 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:20.870893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:20.870895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:20.870971 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.870972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.871675 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.871676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.872611 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:20.872614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.873220 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.873221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:20.873305 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.873319 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:09:20.873383 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:09:20.873424 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
I20260923 23:09:20.873524 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.870467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:46629 (local address 127.5.74.3:39587)
0923 23:09:20.870535 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.870537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.870684 (+ 147us) server_negotiation.cc:407] Connection header received
0923 23:09:20.870764 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.870765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.870775 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.870856 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:20.871052 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.871577 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.872728 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.873126 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.873219 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.873228 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.873414 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.873464 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:09:20.873507 24186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8802c13b2340dfb346300057326589" candidate_uuid: "c3bb25d996424a219219b152722d456b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e7c7f5a354988807f499366471333" is_pre_election: true
I20260923 23:09:20.873728 24186 raft_consensus.cc:2454] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3bb25d996424a219219b152722d456b in term 0.
I20260923 23:09:20.873926 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8802c13b2340dfb346300057326589" candidate_uuid: "c3bb25d996424a219219b152722d456b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd459155b684826be9e4c08e75e1a27" is_pre_election: true
I20260923 23:09:20.874029 24251 leader_election.cc:302] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9e7c7f5a354988807f499366471333, c3bb25d996424a219219b152722d456b; no voters:
I20260923 23:09:20.874078 24448 raft_consensus.cc:2454] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3bb25d996424a219219b152722d456b in term 0.
I20260923 23:09:20.874177 24515 raft_consensus.cc:2781] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:09:20.874281 24515 raft_consensus.cc:482] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:09:20.874322 24515 raft_consensus.cc:3037] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:20.875222 24515 raft_consensus.cc:504] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } }
I20260923 23:09:20.875388 24515 leader_election.cc:288] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [CANDIDATE]: Term 1 election: Requested vote from peers ddd459155b684826be9e4c08e75e1a27 (127.5.74.3:39587), 7a9e7c7f5a354988807f499366471333 (127.5.74.1:42587)
I20260923 23:09:20.875669 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8802c13b2340dfb346300057326589" candidate_uuid: "c3bb25d996424a219219b152722d456b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd459155b684826be9e4c08e75e1a27"
I20260923 23:09:20.875720 24186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8802c13b2340dfb346300057326589" candidate_uuid: "c3bb25d996424a219219b152722d456b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e7c7f5a354988807f499366471333"
I20260923 23:09:20.875773 24448 raft_consensus.cc:3037] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:20.875794 24186 raft_consensus.cc:3037] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:20.876787 24448 raft_consensus.cc:2454] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3bb25d996424a219219b152722d456b in term 1.
I20260923 23:09:20.876787 24186 raft_consensus.cc:2454] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3bb25d996424a219219b152722d456b in term 1.
I20260923 23:09:20.877002 24250 leader_election.cc:302] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3bb25d996424a219219b152722d456b, ddd459155b684826be9e4c08e75e1a27; no voters:
I20260923 23:09:20.877110 24515 raft_consensus.cc:2781] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:20.877362 24515 raft_consensus.cc:686] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 1 LEADER]: Becoming Leader. State: Replica: c3bb25d996424a219219b152722d456b, State: Running, Role: LEADER
I20260923 23:09:20.877506 24515 consensus_queue.cc:237] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } }
I20260923 23:09:20.878593 24044 catalog_manager.cc:5495] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3bb25d996424a219219b152722d456b (127.5.74.2). New cstate: current_term: 1 leader_uuid: "c3bb25d996424a219219b152722d456b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bb25d996424a219219b152722d456b" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 34311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20260923 23:09:20.892963 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.890393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.5.74.2:34311)
0923 23:09:20.890473 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:20.890476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:20.890483 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:09:20.890525 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:20.890526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:20.890546 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:20.890629 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:20.890776 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.891269 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.892042 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:20.892305 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:20.892399 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:20.892575 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:20.892681 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:20.892795 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:20.892827 (+ 32us) server_negotiation.cc:299] Negotiation successful
0923 23:09:20.892836 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 23:09:20.895215 24186 raft_consensus.cc:1261] T 2d8802c13b2340dfb346300057326589 P 7a9e7c7f5a354988807f499366471333 [term 1 FOLLOWER]: Refusing update from remote peer c3bb25d996424a219219b152722d456b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:20.895215 24448 raft_consensus.cc:1261] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [term 1 FOLLOWER]: Refusing update from remote peer c3bb25d996424a219219b152722d456b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:20.895714 24515 consensus_queue.cc:1035] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd459155b684826be9e4c08e75e1a27" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:20.895932 24518 consensus_queue.cc:1035] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9e7c7f5a354988807f499366471333" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:20.899744 24521 mvcc.cc:205] Tried to move back new op lower bound from 7332679519822774272 to 7332679519755055104. Current Snapshot: MvccSnapshot[applied={T|T < 7332679519822774272}]
I20260923 23:09:20.900089 24522 mvcc.cc:205] Tried to move back new op lower bound from 7332679519822774272 to 7332679519755055104. Current Snapshot: MvccSnapshot[applied={T|T < 7332679519822774272}]
I20260923 23:09:20.900656 24524 mvcc.cc:205] Tried to move back new op lower bound from 7332679519822774272 to 7332679519755055104. Current Snapshot: MvccSnapshot[applied={T|T < 7332679519822774272}]
I20260923 23:09:20.903247 24042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:20.903455 24042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:20.904394 24042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51e42530ba814ab3ab966763abeac8c9: Sending DeleteTablet for 3 replicas of tablet 2d8802c13b2340dfb346300057326589
I20260923 23:09:20.904752 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8802c13b2340dfb346300057326589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:20 UTC) from {username='slave'} at 127.0.0.1:45582
I20260923 23:09:20.904812 24166 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8802c13b2340dfb346300057326589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:20 UTC) from {username='slave'} at 127.0.0.1:33186
I20260923 23:09:20.905025 24428 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8802c13b2340dfb346300057326589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:20 UTC) from {username='slave'} at 127.0.0.1:52528
I20260923 23:09:20.905272 24533 tablet_replica.cc:330] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: stopping tablet replica
I20260923 23:09:20.905412 24533 raft_consensus.cc:2229] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:09:20.905532 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 24104
I20260923 23:09:20.905552 24533 raft_consensus.cc:2258] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:20.906039 24533 ts_tablet_manager.cc:1905] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:09:20.907898 24533 ts_tablet_manager.cc:1918] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:20.908003 24533 log.cc:1196] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/2d8802c13b2340dfb346300057326589
I20260923 23:09:20.908314 24533 ts_tablet_manager.cc:1939] T 2d8802c13b2340dfb346300057326589 P ddd459155b684826be9e4c08e75e1a27: Deleting consensus metadata
I20260923 23:09:20.908784 24028 catalog_manager.cc:4841] TS ddd459155b684826be9e4c08e75e1a27 (127.5.74.3:39587): tablet 2d8802c13b2340dfb346300057326589 (table test-table [id=0e666d85b79d4f429b6122c8474f7ec7]) successfully deleted
I20260923 23:09:20.912264 24531 tablet_replica.cc:330] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b: stopping tablet replica
I20260923 23:09:20.912434 24531 raft_consensus.cc:2229] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:09:20.912601 24531 raft_consensus.cc:2258] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:20.913152 24531 ts_tablet_manager.cc:1905] T 2d8802c13b2340dfb346300057326589 P c3bb25d996424a219219b152722d456b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 23:09:20.914305 24029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:42587 (error 108)
I20260923 23:09:20.914570 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 24235
I20260923 23:09:20.915580 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.915418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:42587 (local address 127.0.0.1:33196)
0923 23:09:20.915488 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.915536 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:42587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 23:09:20.915715 24029 catalog_manager.cc:4584] TS 7a9e7c7f5a354988807f499366471333 (127.5.74.1:42587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d8802c13b2340dfb346300057326589: Network error: Client connection negotiation failed: client connection to 127.5.74.1:42587: connect: Connection refused (error 111)
I20260923 23:09:20.919469 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 24366
I20260923 23:09:20.919631 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:20.919475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:34311 (local address 127.0.0.1:45590)
0923 23:09:20.919553 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:20.919589 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.2:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 23:09:20.924507 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 24013
2026-09-23T23:09:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (731 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:20 dist-test-slave-j4c5 krb5kdc[24537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:20 dist-test-slave-j4c5 krb5kdc[24537](info): set up 2 sockets
Sep 23 23:09:20 dist-test-slave-j4c5 krb5kdc[24537](info): commencing operation
krb5kdc: starting...
W20260923 23:09:22.092742 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:22 dist-test-slave-j4c5 krb5kdc[24537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:22.224115 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:45537
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:45537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:22.313282 24550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:22.313557 24550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:22.313586 24550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:22.315145 24550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:22.315214 24550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:22.315232 24550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:22.315248 24550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:22.315261 24550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:22.317224 24550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:45537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:22.317552 24550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:22.317816 24550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:22.323762 24555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:22.323762 24558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:22.323762 24556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:22.324211 24550 server_base.cc:1034] running on GCE node
I20260923 23:09:22.324539 24550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:22.324858 24550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:22.326032 24550 hybrid_clock.cc:648] HybridClock initialized: now 1790204962326014 us; error 31 us; skew 500 ppm
Sep 23 23:09:22 dist-test-slave-j4c5 krb5kdc[24537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:22.329113 24550 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:22.329634 24550 webserver.cc:460] Webserver started at http://127.5.74.62:36727/ using document root <none> and password file <none>
I20260923 23:09:22.329892 24550 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:22.329941 24550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:22.330087 24550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:22.331244 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "11b94fd76ad24ab9b1b022aafad38e5f"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "5a90ac6fc4300b93d39afc14cc8b7427"
server_key_iv: "e63d491e724472ebe0cced1d44a02282"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.331624 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "11b94fd76ad24ab9b1b022aafad38e5f"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "5a90ac6fc4300b93d39afc14cc8b7427"
server_key_iv: "e63d491e724472ebe0cced1d44a02282"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.333212 24550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:22.334163 24565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.334396 24550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:22.334504 24550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "11b94fd76ad24ab9b1b022aafad38e5f"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "5a90ac6fc4300b93d39afc14cc8b7427"
server_key_iv: "e63d491e724472ebe0cced1d44a02282"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.334597 24550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:22.361601 24550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:22.364094 24550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:22.364296 24550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:22.368278 24550 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:45537
I20260923 23:09:22.368320 24617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:45537 every 8 connection(s)
I20260923 23:09:22.368729 24550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:22.369695 24618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:22.370018 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 24550
I20260923 23:09:22.370146 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:22.370368 5416 external_mini_cluster.cc:1428] Setting key 70ba8645ee1a21b9f9b0d63ee6a15e0d
I20260923 23:09:22.373090 24618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f: Bootstrap starting.
I20260923 23:09:22.373737 24618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:22.374079 24618 log.cc:824] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:22.374976 24618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f: No bootstrap required, opened a new log
I20260923 23:09:22.378093 24618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b94fd76ad24ab9b1b022aafad38e5f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45537 } }
I20260923 23:09:22.378281 24618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:22.378307 24618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b94fd76ad24ab9b1b022aafad38e5f, State: Initialized, Role: FOLLOWER
Sep 23 23:09:22 dist-test-slave-j4c5 krb5kdc[24537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:22.378468 24618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b94fd76ad24ab9b1b022aafad38e5f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45537 } }
I20260923 23:09:22.378594 24618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:22.378644 24618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:22.378708 24618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:22.379460 24618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b94fd76ad24ab9b1b022aafad38e5f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45537 } }
I20260923 23:09:22.379567 24618 leader_election.cc:302] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11b94fd76ad24ab9b1b022aafad38e5f; no voters:
I20260923 23:09:22.379873 24618 leader_election.cc:288] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:22.379904 24623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:22.380035 24623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [term 1 LEADER]: Becoming Leader. State: Replica: 11b94fd76ad24ab9b1b022aafad38e5f, State: Running, Role: LEADER
I20260923 23:09:22.380234 24618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:22.380218 24623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b94fd76ad24ab9b1b022aafad38e5f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45537 } }
I20260923 23:09:22.380777 24625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11b94fd76ad24ab9b1b022aafad38e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b94fd76ad24ab9b1b022aafad38e5f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45537 } } }
I20260923 23:09:22.380988 24625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:22.381299 24632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:22.381728 24624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11b94fd76ad24ab9b1b022aafad38e5f. Latest consensus state: current_term: 1 leader_uuid: "11b94fd76ad24ab9b1b022aafad38e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b94fd76ad24ab9b1b022aafad38e5f" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45537 } } }
I20260923 23:09:22.381943 24624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:22.382172 24632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:22.384070 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:22.370942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.5.74.62:45537)
0923 23:09:22.371163 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:22.371176 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:22.371203 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:09:22.371651 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:22.371658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:22.371707 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:22.371835 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:22.372747 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.373964 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.375004 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.375570 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.375672 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:22.379163 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:22.379167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:22.379168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:22.379180 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:22.381742 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:22.382285 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:22.382289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:22.382290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:22.382420 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:22.382758 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:22.382759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:22.382760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:22.383065 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:22.383257 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:22.383587 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 23:09:22.383776 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260923 23:09:22.388131 24632 catalog_manager.cc:1347] Generated new cluster ID: d1b4df495b1543c3a3d9688abc47f88d
I20260923 23:09:22.388265 24632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:22.405792 24632 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:22.406709 24632 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:22.413085 24632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f: Generated new TSK 0
I20260923 23:09:22.413359 24632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.74.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
Entry for principal kudu/127.5.74.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab.
I20260923 23:09:22.447445 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:45537
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:22.542632 24644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:22.542873 24644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:22.542901 24644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:22.544340 24644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:22.544397 24644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:22.544437 24644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:09:22.546198 24644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.1.keytab
--principal=kudu/127.5.74.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:22.546511 24644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:22.546789 24644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:22.552790 24649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:22.552790 24652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:22.552862 24650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:22.553689 24644 server_base.cc:1034] running on GCE node
I20260923 23:09:22.553954 24644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:22.554195 24644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:22.555351 24644 hybrid_clock.cc:648] HybridClock initialized: now 1790204962555345 us; error 47 us; skew 500 ppm
Sep 23 23:09:22 dist-test-slave-j4c5 krb5kdc[24537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:22.558215 24644 init.cc:377] Logged in from keytab as kudu/127.5.74.1@KRBTEST.COM (short username kudu)
I20260923 23:09:22.558727 24644 webserver.cc:460] Webserver started at http://127.5.74.1:42267/ using document root <none> and password file <none>
I20260923 23:09:22.558981 24644 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:22.559049 24644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:22.559197 24644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:22.560370 24644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "b1a077489d6c47a8beec773b6a071b9d"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "9b7a8052d22950b135ef2e85a2097a69"
server_key_iv: "e8eb874386de3719c30a6ce4165c88ac"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.560741 24644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "b1a077489d6c47a8beec773b6a071b9d"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "9b7a8052d22950b135ef2e85a2097a69"
server_key_iv: "e8eb874386de3719c30a6ce4165c88ac"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.562381 24644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:09:22.563337 24659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.563695 24644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:22.563807 24644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "b1a077489d6c47a8beec773b6a071b9d"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "9b7a8052d22950b135ef2e85a2097a69"
server_key_iv: "e8eb874386de3719c30a6ce4165c88ac"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.563889 24644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:22.606120 24644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:22.608884 24644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:22.609295 24644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:22.609607 24644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:22.610219 24644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:22.610301 24644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.610349 24644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:22.610378 24644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.617317 24644 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:42697
I20260923 23:09:22.617357 24772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:42697 every 8 connection(s)
I20260923 23:09:22.617722 24644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:09:22 dist-test-slave-j4c5 krb5kdc[24537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.1@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:22.624914 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 24644
I20260923 23:09:22.625032 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:09:22.625270 5416 external_mini_cluster.cc:1428] Setting key b150aa78f8037a9b1fc504af88235043
I20260923 23:09:22.629288 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:22.618657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:43757 (local address 127.5.74.62:45537)
0923 23:09:22.618802 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:22.618805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:22.619004 (+ 199us) server_negotiation.cc:407] Connection header received
0923 23:09:22.619591 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:22.619593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:22.619606 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:22.619665 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:22.620788 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.621302 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.622366 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.622597 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.622687 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:22.625259 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:22.625260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:22.625261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:22.625270 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:22.627444 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:22.628065 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:22.628067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:22.628067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:22.628182 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:22.628450 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:22.628452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:22.628452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:22.628632 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:22.628729 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:22.629089 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 23:09:22.629188 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260923 23:09:22.629590 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:22.618554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:45537 (local address 127.5.74.1:43757)
0923 23:09:22.618864 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:22.618905 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:22.619477 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:22.619749 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:22.619754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:22.620463 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:22.620675 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:22.620681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.621420 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.621421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:22.622263 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:22.622265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.622704 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.622705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:22.622843 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:22.623410 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:22.623419 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:22.625155 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:22.627633 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:22.627646 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:22.627650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:22.627974 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:22.628266 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:22.628266 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:22.628267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:22.628381 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:22.628758 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:22.628764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:22.628909 (+ 145us) client_negotiation.cc:769] Sending connection context
0923 23:09:22.629062 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 23:09:22.629251 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":125,"threads_started":1}
I20260923 23:09:22.630249 24773 heartbeater.cc:347] Connected to a master server at 127.5.74.62:45537
I20260923 23:09:22.630385 24773 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:22.630774 24773 heartbeater.cc:511] Master 127.5.74.62:45537 requested a full tablet report, sending...
I20260923 23:09:22.631590 24582 ts_manager.cc:194] Registered new tserver with Master: b1a077489d6c47a8beec773b6a071b9d (127.5.74.1:42697)
I20260923 23:09:22.632689 24582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.1@KRBTEST.COM'} at 127.5.74.1:43757
WARNING: no policy specified for kudu/127.5.74.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
Entry for principal kudu/127.5.74.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab.
I20260923 23:09:22.656862 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:45537
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:22.743734 24778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:22.743966 24778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:22.743995 24778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:22.745175 24778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:22.745218 24778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:22.745246 24778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:09:22.746697 24778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.2.keytab
--principal=kudu/127.5.74.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:22.746917 24778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:22.747164 24778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:22.752970 24784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:22.752970 24786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:22.752965 24783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:22.753629 24778 server_base.cc:1034] running on GCE node
I20260923 23:09:22.753895 24778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:22.754134 24778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:22.755306 24778 hybrid_clock.cc:648] HybridClock initialized: now 1790204962755284 us; error 44 us; skew 500 ppm
Sep 23 23:09:22 dist-test-slave-j4c5 krb5kdc[24537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:22.758143 24778 init.cc:377] Logged in from keytab as kudu/127.5.74.2@KRBTEST.COM (short username kudu)
I20260923 23:09:22.758639 24778 webserver.cc:460] Webserver started at http://127.5.74.2:42857/ using document root <none> and password file <none>
I20260923 23:09:22.758886 24778 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:22.758978 24778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:22.759123 24778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:22.760329 24778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "342175598a5d4b36b0eabd9e5061f8b5"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "200fc64bad13374e6326ae8a6a20a7c0"
server_key_iv: "d3f9982be73ad8e2fded97ba0c428d65"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.760703 24778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "342175598a5d4b36b0eabd9e5061f8b5"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "200fc64bad13374e6326ae8a6a20a7c0"
server_key_iv: "d3f9982be73ad8e2fded97ba0c428d65"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.762100 24778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:09:22.762998 24793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.763247 24778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:22.763330 24778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "342175598a5d4b36b0eabd9e5061f8b5"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "200fc64bad13374e6326ae8a6a20a7c0"
server_key_iv: "d3f9982be73ad8e2fded97ba0c428d65"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.763382 24778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:22.813127 24778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:22.815680 24778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:22.815858 24778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:22.816200 24778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:22.816546 24778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:22.816603 24778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.816642 24778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:22.816671 24778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.822293 24778 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:43803
I20260923 23:09:22.822373 24906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:43803 every 8 connection(s)
I20260923 23:09:22.822714 24778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:09:22.822856 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 24778
I20260923 23:09:22.822997 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:09:22.823241 5416 external_mini_cluster.cc:1428] Setting key 0a25ec6187391d64490c84a0400a8dea
Sep 23 23:09:22 dist-test-slave-j4c5 krb5kdc[24537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.2@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.74.3@KRBTEST.COM; defaulting to no policy
I20260923 23:09:22.834293 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:22.823636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:50877 (local address 127.5.74.62:45537)
0923 23:09:22.823686 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:22.823688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:22.824181 (+ 493us) server_negotiation.cc:407] Connection header received
0923 23:09:22.824635 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:22.824636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:22.824649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:22.824723 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:22.825847 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.826398 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.828017 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.828375 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.828440 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:22.831352 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:22.831354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:22.831354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:22.831364 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:22.832966 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:22.833427 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:22.833428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:22.833428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:22.833489 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:22.833733 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:22.833733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:22.833733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:22.833839 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:22.833890 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:22.834157 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 23:09:22.834225 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:09:22.834470 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:22.823561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:45537 (local address 127.5.74.2:50877)
0923 23:09:22.823876 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:22.823981 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:22.824533 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:22.824775 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:22.824778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:22.825371 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:22.825706 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:22.825713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.826514 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.826515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:22.827899 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:22.827902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.828470 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.828471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:22.828633 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:22.829606 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:22.829618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:22.831238 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:22.833055 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:22.833062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:22.833065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:22.833310 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:22.833562 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:22.833563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:22.833563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:22.833672 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:22.833894 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:22.833898 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:22.834034 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 23:09:22.834122 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 23:09:22.834251 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":84,"threads_started":1}
I20260923 23:09:22.834944 24907 heartbeater.cc:347] Connected to a master server at 127.5.74.62:45537
I20260923 23:09:22.835040 24907 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.3@KRBTEST.COM" created.
I20260923 23:09:22.835246 24907 heartbeater.cc:511] Master 127.5.74.62:45537 requested a full tablet report, sending...
I20260923 23:09:22.835742 24582 ts_manager.cc:194] Registered new tserver with Master: 342175598a5d4b36b0eabd9e5061f8b5 (127.5.74.2:43803)
I20260923 23:09:22.836143 24582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.2@KRBTEST.COM'} at 127.5.74.2:50877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
Entry for principal kudu/127.5.74.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab.
I20260923 23:09:22.852391 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:45537
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:22.936902 24912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:22.937158 24912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:22.937202 24912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:22.938513 24912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:22.938592 24912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:22.938638 24912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:09:22.940236 24912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.3.keytab
--principal=kudu/127.5.74.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:22.940528 24912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:22.940776 24912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:22.946643 24920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:22.946668 24912 server_base.cc:1034] running on GCE node
W20260923 23:09:22.946630 24918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:22.946630 24917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:22.947234 24912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:22.947535 24912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:22.948732 24912 hybrid_clock.cc:648] HybridClock initialized: now 1790204962948620 us; error 104 us; skew 500 ppm
Sep 23 23:09:22 dist-test-slave-j4c5 krb5kdc[24537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:22.951929 24912 init.cc:377] Logged in from keytab as kudu/127.5.74.3@KRBTEST.COM (short username kudu)
I20260923 23:09:22.952571 24912 webserver.cc:460] Webserver started at http://127.5.74.3:38477/ using document root <none> and password file <none>
I20260923 23:09:22.952836 24912 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:22.952915 24912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:22.953090 24912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:22.954315 24912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "87b4ff2523ef4721876fab5205ef3711"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "31d38d879f765ad27e97845dd7b67096"
server_key_iv: "cb4f537b24227c16a8b9facaa2cf42ba"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.954757 24912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "87b4ff2523ef4721876fab5205ef3711"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "31d38d879f765ad27e97845dd7b67096"
server_key_iv: "cb4f537b24227c16a8b9facaa2cf42ba"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.956589 24912 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260923 23:09:22.958012 24927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.958273 24912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:22.958416 24912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "87b4ff2523ef4721876fab5205ef3711"
format_stamp: "Formatted at 2026-09-23 23:09:22 on dist-test-slave-j4c5"
server_key: "31d38d879f765ad27e97845dd7b67096"
server_key_iv: "cb4f537b24227c16a8b9facaa2cf42ba"
server_key_version: "encryptionkey@0"
I20260923 23:09:22.958503 24912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:22.975967 24912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:22.978514 24912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:22.978740 24912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:22.978991 24912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:22.979321 24912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:22.979375 24912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.979418 24912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:22.979446 24912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:22.986099 24912 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:39245
I20260923 23:09:22.986141 25040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:39245 every 8 connection(s)
I20260923 23:09:22.986503 24912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:09:22.988116 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 24912
I20260923 23:09:22.988260 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:09:22.988493 5416 external_mini_cluster.cc:1428] Setting key 1bf9a7adb55c70f854bdae77fd9c5abc
Sep 23 23:09:22 dist-test-slave-j4c5 krb5kdc[24537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.3@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:22.997258 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:22.987268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:51077 (local address 127.5.74.62:45537)
0923 23:09:22.987341 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:22.987343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:22.987569 (+ 226us) server_negotiation.cc:407] Connection header received
0923 23:09:22.988093 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:22.988095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:22.988111 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:22.988286 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:22.989608 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.990131 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.991180 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.991392 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.991489 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:22.994071 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:22.994073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:22.994073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:22.994088 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:22.995698 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:22.996228 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:22.996229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:22.996230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:22.996296 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:22.996575 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:22.996576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:22.996576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:22.996703 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:22.996784 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:22.997068 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 23:09:22.997160 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 23:09:22.997484 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:22.987220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:45537 (local address 127.5.74.3:51077)
0923 23:09:22.987405 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:22.987453 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:22.987969 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:22.988405 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:22.988408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:22.989224 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:22.989495 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:22.989501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.990241 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.990242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:22.991083 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:22.991086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:22.991500 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:22.991501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:22.991629 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:22.992250 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:22.992259 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:22.993961 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:22.995830 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:22.995839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:22.995842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:22.996104 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:22.996386 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:22.996387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:22.996388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:22.996506 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:22.996772 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:22.996779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:22.996932 (+ 153us) client_negotiation.cc:769] Sending connection context
0923 23:09:22.997045 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 23:09:22.997221 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260923 23:09:22.998034 25041 heartbeater.cc:347] Connected to a master server at 127.5.74.62:45537
I20260923 23:09:22.998122 25041 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:22.998311 25041 heartbeater.cc:511] Master 127.5.74.62:45537 requested a full tablet report, sending...
I20260923 23:09:22.998803 24582 ts_manager.cc:194] Registered new tserver with Master: 87b4ff2523ef4721876fab5205ef3711 (127.5.74.3:39245)
I20260923 23:09:22.999202 24582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.74.3@KRBTEST.COM'} at 127.5.74.3:51077
I20260923 23:09:23.000936 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:09:23.010489 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.002686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.5.74.62:45537)
0923 23:09:23.002766 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.002769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.002797 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:09:23.002920 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.002921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.002931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.003096 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:23.003753 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.004524 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.005339 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.005725 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.005802 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.007174 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:23.007176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:23.007176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:23.007184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:23.008949 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:23.009450 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:23.009451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:23.009451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:23.009507 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:23.009785 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:23.009787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:23.009787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:23.009987 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:23.010075 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.010278 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.010411 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 23:09:23.012733 24582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:09:23.014681 24582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:09:23.022295 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.017804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:39245 (local address 127.0.0.1:44026)
0923 23:09:23.018568 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.018578 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.018627 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.018936 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.018937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.018939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.019003 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.019005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.020111 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.020112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.021271 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.021274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.021979 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.021980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.022060 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.022077 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.022153 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.022209 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":322,"thread_start_us":83,"threads_started":1}
I20260923 23:09:23.022306 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.018062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.5.74.3:39245)
0923 23:09:23.018249 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.018254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.018700 (+ 446us) server_negotiation.cc:407] Connection header received
0923 23:09:23.018748 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.018750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.018766 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.018848 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.019094 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.019939 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.021367 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.021887 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.021980 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.022057 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.022169 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.022208 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260923 23:09:23.022658 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.017804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:42697 (local address 127.0.0.1:55208)
0923 23:09:23.018240 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.018359 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.018430 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.018799 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.018802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.018805 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.018904 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.018906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.020327 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.020328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.021464 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.021467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.022427 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.022428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.022493 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.022499 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.022532 (+ 33us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.022574 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260923 23:09:23.023142 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.017970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.5.74.1:42697)
0923 23:09:23.018390 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.018395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.018410 (+ 15us) server_negotiation.cc:407] Connection header received
0923 23:09:23.018647 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.018650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.018669 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.018835 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.018990 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.020192 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.021579 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.022319 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.022903 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.022968 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.022995 (+ 27us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.023045 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":133,"threads_started":1}
I20260923 23:09:23.023207 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.017899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:43803 (local address 127.0.0.1:34774)
0923 23:09:23.018277 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.018295 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.018434 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.019464 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.019465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.019466 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.019522 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.019524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.020878 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.020879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.021940 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.021943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.022955 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.022956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.023029 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.023033 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.023090 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.023127 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":91,"threads_started":1}
I20260923 23:09:23.023528 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.018164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.5.74.2:43803)
0923 23:09:23.019171 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.019176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.019189 (+ 13us) server_negotiation.cc:407] Connection header received
0923 23:09:23.019230 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.019232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.019248 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.019373 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.019609 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.020734 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.022046 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.022785 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.022860 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.022918 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.023397 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.023446 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":94,"threads_started":1}
I20260923 23:09:23.023506 24975 tablet_service.cc:1467] Processing CreateTablet for tablet a938525602364933b7fc45c0d9a3b3fd (DEFAULT_TABLE table=test-table [id=e2d53d8de56844bab913d5e43642c74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:23.023761 24707 tablet_service.cc:1467] Processing CreateTablet for tablet a938525602364933b7fc45c0d9a3b3fd (DEFAULT_TABLE table=test-table [id=e2d53d8de56844bab913d5e43642c74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:23.023912 24975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a938525602364933b7fc45c0d9a3b3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:23.024117 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a938525602364933b7fc45c0d9a3b3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:23.024266 24841 tablet_service.cc:1467] Processing CreateTablet for tablet a938525602364933b7fc45c0d9a3b3fd (DEFAULT_TABLE table=test-table [id=e2d53d8de56844bab913d5e43642c74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:23.024613 24841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a938525602364933b7fc45c0d9a3b3fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:23.028061 25059 tablet_bootstrap.cc:492] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d: Bootstrap starting.
I20260923 23:09:23.028363 25058 tablet_bootstrap.cc:492] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: Bootstrap starting.
I20260923 23:09:23.028594 25060 tablet_bootstrap.cc:492] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5: Bootstrap starting.
I20260923 23:09:23.028968 25059 tablet_bootstrap.cc:654] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:23.029251 25058 tablet_bootstrap.cc:654] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:23.029703 25060 tablet_bootstrap.cc:654] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:23.029953 25059 log.cc:824] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:23.030055 25058 log.cc:824] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:23.030365 25060 log.cc:824] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:23.030779 25059 tablet_bootstrap.cc:492] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d: No bootstrap required, opened a new log
I20260923 23:09:23.030871 25059 ts_tablet_manager.cc:1397] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260923 23:09:23.031107 25060 tablet_bootstrap.cc:492] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5: No bootstrap required, opened a new log
I20260923 23:09:23.031162 25060 ts_tablet_manager.cc:1397] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 23:09:23.032648 25058 tablet_bootstrap.cc:492] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: No bootstrap required, opened a new log
I20260923 23:09:23.032732 25058 ts_tablet_manager.cc:1397] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 23:09:23.032691 25060 raft_consensus.cc:348] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } }
I20260923 23:09:23.032727 25059 raft_consensus.cc:348] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } }
I20260923 23:09:23.032925 25059 raft_consensus.cc:374] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:23.032927 25060 raft_consensus.cc:374] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:23.032970 25060 raft_consensus.cc:729] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342175598a5d4b36b0eabd9e5061f8b5, State: Initialized, Role: FOLLOWER
I20260923 23:09:23.032970 25059 raft_consensus.cc:729] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a077489d6c47a8beec773b6a071b9d, State: Initialized, Role: FOLLOWER
I20260923 23:09:23.033085 25060 consensus_queue.cc:260] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } }
I20260923 23:09:23.033087 25059 consensus_queue.cc:260] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } }
I20260923 23:09:23.033402 25060 ts_tablet_manager.cc:1428] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:09:23.033396 25059 ts_tablet_manager.cc:1428] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 23:09:23.033438 24907 heartbeater.cc:503] Master 127.5.74.62:45537 was elected leader, sending a full tablet report...
I20260923 23:09:23.033545 24773 heartbeater.cc:503] Master 127.5.74.62:45537 was elected leader, sending a full tablet report...
I20260923 23:09:23.035239 25058 raft_consensus.cc:348] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } }
I20260923 23:09:23.035398 25058 raft_consensus.cc:374] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:23.035420 25058 raft_consensus.cc:729] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b4ff2523ef4721876fab5205ef3711, State: Initialized, Role: FOLLOWER
I20260923 23:09:23.035548 25058 consensus_queue.cc:260] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } }
I20260923 23:09:23.035849 25058 ts_tablet_manager.cc:1428] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:09:23.035939 25041 heartbeater.cc:503] Master 127.5.74.62:45537 was elected leader, sending a full tablet report...
W20260923 23:09:23.073647 24908 tablet.cc:2369] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:09:23.118535 24774 tablet.cc:2369] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:09:23.124102 25066 raft_consensus.cc:482] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:23.124308 25066 raft_consensus.cc:504] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } }
I20260923 23:09:23.124754 25066 leader_election.cc:288] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1a077489d6c47a8beec773b6a071b9d (127.5.74.1:42697), 342175598a5d4b36b0eabd9e5061f8b5 (127.5.74.2:43803)
I20260923 23:09:23.128834 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.125384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:35051 (local address 127.5.74.2:43803)
0923 23:09:23.125443 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.125446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.125456 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:09:23.125486 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.125487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.125501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.125554 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.125788 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.126546 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.127642 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.128353 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.128429 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.128441 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.128667 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.128715 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:09:23.128947 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.124935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:42697 (local address 127.5.74.3:38209)
0923 23:09:23.125059 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.125073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.125132 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.125408 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.125409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.125411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.125488 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.125490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.126551 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.126553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.127777 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.127780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.128674 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.128675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.128750 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.128752 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.128803 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.128861 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260923 23:09:23.129073 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.125004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:43803 (local address 127.5.74.3:35051)
0923 23:09:23.125252 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.125265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.125330 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.125610 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.125611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.125613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.125693 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.125695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.126621 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.126623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.127515 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.127518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.128479 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.128481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.128568 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.128574 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.128938 (+ 364us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.128984 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":190,"threads_started":1}
I20260923 23:09:23.129273 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.125085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:38209 (local address 127.5.74.1:42697)
0923 23:09:23.125222 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.125225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.125234 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:09:23.125265 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.125266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.125280 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.125336 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.125608 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.126463 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.127884 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.128559 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.128632 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.128644 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.129166 (+ 522us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.129222 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260923 23:09:23.129618 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a938525602364933b7fc45c0d9a3b3fd" candidate_uuid: "87b4ff2523ef4721876fab5205ef3711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a077489d6c47a8beec773b6a071b9d" is_pre_election: true
I20260923 23:09:23.129751 24727 raft_consensus.cc:2454] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87b4ff2523ef4721876fab5205ef3711 in term 0.
I20260923 23:09:23.130116 24930 leader_election.cc:302] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b4ff2523ef4721876fab5205ef3711, b1a077489d6c47a8beec773b6a071b9d; no voters:
I20260923 23:09:23.130244 25066 raft_consensus.cc:2781] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:09:23.130293 25066 raft_consensus.cc:482] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:09:23.130322 25066 raft_consensus.cc:3037] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:23.131188 25066 raft_consensus.cc:504] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } }
I20260923 23:09:23.131338 25066 leader_election.cc:288] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [CANDIDATE]: Term 1 election: Requested vote from peers b1a077489d6c47a8beec773b6a071b9d (127.5.74.1:42697), 342175598a5d4b36b0eabd9e5061f8b5 (127.5.74.2:43803)
I20260923 23:09:23.131526 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a938525602364933b7fc45c0d9a3b3fd" candidate_uuid: "87b4ff2523ef4721876fab5205ef3711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a077489d6c47a8beec773b6a071b9d"
I20260923 23:09:23.131603 24727 raft_consensus.cc:3037] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:23.132419 24727 raft_consensus.cc:2454] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b4ff2523ef4721876fab5205ef3711 in term 1.
I20260923 23:09:23.136914 24930 leader_election.cc:302] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b4ff2523ef4721876fab5205ef3711, b1a077489d6c47a8beec773b6a071b9d; no voters:
I20260923 23:09:23.136845 24860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a938525602364933b7fc45c0d9a3b3fd" candidate_uuid: "87b4ff2523ef4721876fab5205ef3711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342175598a5d4b36b0eabd9e5061f8b5"
I20260923 23:09:23.136828 24861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a938525602364933b7fc45c0d9a3b3fd" candidate_uuid: "87b4ff2523ef4721876fab5205ef3711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342175598a5d4b36b0eabd9e5061f8b5" is_pre_election: true
I20260923 23:09:23.137065 25066 raft_consensus.cc:2781] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:23.137051 24860 raft_consensus.cc:3037] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:23.137243 25066 raft_consensus.cc:686] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 1 LEADER]: Becoming Leader. State: Replica: 87b4ff2523ef4721876fab5205ef3711, State: Running, Role: LEADER
I20260923 23:09:23.137342 25066 consensus_queue.cc:237] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } }
I20260923 23:09:23.138060 24860 raft_consensus.cc:2454] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b4ff2523ef4721876fab5205ef3711 in term 1.
I20260923 23:09:23.138303 24581 catalog_manager.cc:5495] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87b4ff2523ef4721876fab5205ef3711 (127.5.74.3). New cstate: current_term: 1 leader_uuid: "87b4ff2523ef4721876fab5205ef3711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87b4ff2523ef4721876fab5205ef3711" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 39245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 } health_report { overall_health: UNKNOWN } } }
I20260923 23:09:23.170164 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.167556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.5.74.3:39245)
0923 23:09:23.167606 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.167608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.167639 (+ 31us) server_negotiation.cc:407] Connection header received
0923 23:09:23.167723 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.167724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.167744 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.167819 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:23.167976 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.168540 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.169344 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.169569 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.169627 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.169812 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:23.169941 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:23.170009 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.170059 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.170080 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:09:23.172209 24860 raft_consensus.cc:1261] T a938525602364933b7fc45c0d9a3b3fd P 342175598a5d4b36b0eabd9e5061f8b5 [term 1 FOLLOWER]: Refusing update from remote peer 87b4ff2523ef4721876fab5205ef3711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:23.172273 24727 raft_consensus.cc:1261] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [term 1 FOLLOWER]: Refusing update from remote peer 87b4ff2523ef4721876fab5205ef3711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:23.172641 25068 consensus_queue.cc:1035] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a077489d6c47a8beec773b6a071b9d" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 42697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:23.172796 25066 consensus_queue.cc:1035] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342175598a5d4b36b0eabd9e5061f8b5" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 43803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:23.175473 25071 mvcc.cc:205] Tried to move back new op lower bound from 7332679529150013440 to 7332679529011036160. Current Snapshot: MvccSnapshot[applied={T|T < 7332679529150013440}]
I20260923 23:09:23.175624 25073 mvcc.cc:205] Tried to move back new op lower bound from 7332679529150013440 to 7332679529011036160. Current Snapshot: MvccSnapshot[applied={T|T < 7332679529150013440}]
I20260923 23:09:23.178709 24581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:23.178813 24581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:23.179610 24581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11b94fd76ad24ab9b1b022aafad38e5f: Sending DeleteTablet for 3 replicas of tablet a938525602364933b7fc45c0d9a3b3fd
I20260923 23:09:23.179955 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet a938525602364933b7fc45c0d9a3b3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:23 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:55208
I20260923 23:09:23.180015 24975 tablet_service.cc:1514] Processing DeleteTablet for tablet a938525602364933b7fc45c0d9a3b3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:23 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:44026
I20260923 23:09:23.179976 24841 tablet_service.cc:1514] Processing DeleteTablet for tablet a938525602364933b7fc45c0d9a3b3fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:23 UTC) from {username='kudu', principal='kudu/127.5.74.62@KRBTEST.COM'} at 127.0.0.1:34774
I20260923 23:09:23.180241 25081 tablet_replica.cc:330] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d: stopping tablet replica
I20260923 23:09:23.180619 25081 raft_consensus.cc:2229] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:09:23.180814 25081 raft_consensus.cc:2258] T a938525602364933b7fc45c0d9a3b3fd P b1a077489d6c47a8beec773b6a071b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:23.180963 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 24644
I20260923 23:09:23.181056 25082 tablet_replica.cc:330] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: stopping tablet replica
I20260923 23:09:23.182900 25082 raft_consensus.cc:2229] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:09:23.183063 25082 raft_consensus.cc:2258] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:23.183700 25082 ts_tablet_manager.cc:1905] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:09:23.185988 25082 ts_tablet_manager.cc:1918] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:23.186117 25082 log.cc:1196] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/a938525602364933b7fc45c0d9a3b3fd
I20260923 23:09:23.186465 25082 ts_tablet_manager.cc:1939] T a938525602364933b7fc45c0d9a3b3fd P 87b4ff2523ef4721876fab5205ef3711: Deleting consensus metadata
W20260923 23:09:23.186794 24568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:42697 (error 108)
I20260923 23:09:23.187000 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 24778
I20260923 23:09:23.192210 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.192020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:43803 (local address 127.0.0.1:34790)
0923 23:09:23.192082 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.192134 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.2:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 23:09:23.192337 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.192104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:42697 (local address 127.0.0.1:55210)
0923 23:09:23.192190 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.192218 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 23:09:23.192436 24569 catalog_manager.cc:4584] TS 342175598a5d4b36b0eabd9e5061f8b5 (127.5.74.2:43803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a938525602364933b7fc45c0d9a3b3fd: Network error: Client connection negotiation failed: client connection to 127.5.74.2:43803: connect: Connection refused (error 111)
I20260923 23:09:23.192628 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 24912
I20260923 23:09:23.197427 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 24550
I20260923 23:09:23.197654 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.197419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:39245 (local address 127.0.0.1:44038)
0923 23:09:23.197505 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.197609 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.3:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
2026-09-23T23:09:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2273 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T23:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:23Z Disabled control of system clock
I20260923 23:09:23.227648 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:43199
--webserver_interface=127.5.74.62
--webserver_port=0
--builtin_ntp_servers=127.5.74.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:43199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:09:23.316763 25088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:23.317055 25088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:23.317094 25088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:23.318445 25088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:23.318521 25088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:23.318542 25088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:23.318572 25088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:23.318599 25088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:23.320217 25088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:43199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:43199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:23.320538 25088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:23.320856 25088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:23.326841 25093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:23.326810 25094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:23.326805 25096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:23.327147 25088 server_base.cc:1034] running on GCE node
I20260923 23:09:23.327517 25088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:23.327931 25088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:23.329119 25088 hybrid_clock.cc:648] HybridClock initialized: now 1790204963329078 us; error 62 us; skew 500 ppm
I20260923 23:09:23.330600 25088 webserver.cc:460] Webserver started at http://127.5.74.62:39275/ using document root <none> and password file <none>
I20260923 23:09:23.330883 25088 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:23.330961 25088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:23.331126 25088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:23.332391 25088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "f728bf7e47f8452381e5eeee61d99588"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "5b4d79201d147112d2b2fe79431f331b"
server_key_iv: "e334f4c530277c38f681680bea624c2a"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.332814 25088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "f728bf7e47f8452381e5eeee61d99588"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "5b4d79201d147112d2b2fe79431f331b"
server_key_iv: "e334f4c530277c38f681680bea624c2a"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.334254 25088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:09:23.335451 25102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.335716 25088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:23.335847 25088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "f728bf7e47f8452381e5eeee61d99588"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "5b4d79201d147112d2b2fe79431f331b"
server_key_iv: "e334f4c530277c38f681680bea624c2a"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.335950 25088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:23.350687 25088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:23.351102 25088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:23.351275 25088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:23.355592 25088 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:43199
I20260923 23:09:23.355693 25154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:43199 every 8 connection(s)
I20260923 23:09:23.356061 25088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:23.357084 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:23.359396 25155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588: Bootstrap starting.
I20260923 23:09:23.359967 25155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:23.360329 25155 log.cc:824] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:23.361047 25155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588: No bootstrap required, opened a new log
I20260923 23:09:23.362645 25155 raft_consensus.cc:348] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f728bf7e47f8452381e5eeee61d99588" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43199 } }
I20260923 23:09:23.362795 25155 raft_consensus.cc:374] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:23.362818 25155 raft_consensus.cc:729] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f728bf7e47f8452381e5eeee61d99588, State: Initialized, Role: FOLLOWER
I20260923 23:09:23.362963 25155 consensus_queue.cc:260] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f728bf7e47f8452381e5eeee61d99588" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43199 } }
I20260923 23:09:23.363050 25155 raft_consensus.cc:388] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:23.363072 25155 raft_consensus.cc:482] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:23.363101 25155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:23.363372 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 25088
I20260923 23:09:23.363513 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:23.363746 5416 external_mini_cluster.cc:1428] Setting key 7167530a373e5b38f898d45369351931
I20260923 23:09:23.363804 25155 raft_consensus.cc:504] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f728bf7e47f8452381e5eeee61d99588" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43199 } }
I20260923 23:09:23.363941 25155 leader_election.cc:302] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f728bf7e47f8452381e5eeee61d99588; no voters:
I20260923 23:09:23.364176 25155 leader_election.cc:288] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:23.364243 25158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:23.364538 25155 sys_catalog.cc:564] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:23.364619 25158 raft_consensus.cc:686] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [term 1 LEADER]: Becoming Leader. State: Replica: f728bf7e47f8452381e5eeee61d99588, State: Running, Role: LEADER
I20260923 23:09:23.364758 25158 consensus_queue.cc:237] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f728bf7e47f8452381e5eeee61d99588" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43199 } }
I20260923 23:09:23.365110 25158 sys_catalog.cc:455] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f728bf7e47f8452381e5eeee61d99588. Latest consensus state: current_term: 1 leader_uuid: "f728bf7e47f8452381e5eeee61d99588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f728bf7e47f8452381e5eeee61d99588" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43199 } } }
I20260923 23:09:23.365231 25162 sys_catalog.cc:455] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f728bf7e47f8452381e5eeee61d99588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f728bf7e47f8452381e5eeee61d99588" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 43199 } } }
I20260923 23:09:23.365233 25158 sys_catalog.cc:458] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:23.365317 25162 sys_catalog.cc:458] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:23.365576 25169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:23.366142 25169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:23.367961 25169 catalog_manager.cc:1347] Generated new cluster ID: e418d56188e44747afa1994f768f46c9
I20260923 23:09:23.368024 25169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:23.370350 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.364438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.5.74.62:43199)
0923 23:09:23.364674 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.364684 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.364723 (+ 39us) server_negotiation.cc:407] Connection header received
0923 23:09:23.364849 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.364857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.364930 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.365384 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:23.365511 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.366972 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.368140 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.368647 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.368799 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.369567 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:23.369572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:23.369572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:23.369777 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:23.369813 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:23.369867 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:23.369958 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.370066 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.370157 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":79,"threads_started":1}
I20260923 23:09:23.378741 25169 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:23.379313 25169 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:23.382968 25169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588: Generated new TSK 0
I20260923 23:09:23.383199 25169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:23.387353 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.1:0
--local_ip_for_outbound_sockets=127.5.74.1
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43199
--builtin_ntp_servers=127.5.74.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:09:23.495551 25179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:23.495781 25179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:23.495808 25179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:23.497269 25179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:23.497332 25179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:23.497362 25179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.1
I20260923 23:09:23.499081 25179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.74.1
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.1
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:23.499396 25179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:23.499773 25179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:23.506636 25185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:23.506767 25187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:23.506775 25184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:23.506979 25179 server_base.cc:1034] running on GCE node
I20260923 23:09:23.507157 25179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:23.507433 25179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:23.508591 25179 hybrid_clock.cc:648] HybridClock initialized: now 1790204963508569 us; error 34 us; skew 500 ppm
I20260923 23:09:23.510248 25179 webserver.cc:460] Webserver started at http://127.5.74.1:43511/ using document root <none> and password file <none>
I20260923 23:09:23.510517 25179 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:23.510564 25179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:23.510663 25179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:23.511747 25179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/data/instance:
uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "46bb4720023c0dbbad559f01c79d22fd"
server_key_iv: "e1727b73e50e5cea354169a31229d787"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.512050 25179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance:
uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "46bb4720023c0dbbad559f01c79d22fd"
server_key_iv: "e1727b73e50e5cea354169a31229d787"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.513480 25179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:23.514249 25193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.514498 25179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:23.514595 25179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/wal
uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "46bb4720023c0dbbad559f01c79d22fd"
server_key_iv: "e1727b73e50e5cea354169a31229d787"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.514657 25179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:23.534197 25179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:23.534565 25179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:23.534740 25179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:23.534951 25179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:23.535282 25179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:23.535315 25179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.535368 25179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:23.535394 25179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.541973 25179 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.1:38859
I20260923 23:09:23.542022 25306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.1:38859 every 8 connection(s)
I20260923 23:09:23.542389 25179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/data/info.pb
I20260923 23:09:23.544901 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 25179
I20260923 23:09:23.545008 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-0/wal/instance
I20260923 23:09:23.545241 5416 external_mini_cluster.cc:1428] Setting key 6c916d0a28162791877fb52bedb708d7
I20260923 23:09:23.546370 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.2:0
--local_ip_for_outbound_sockets=127.5.74.2
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43199
--builtin_ntp_servers=127.5.74.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:09:23.548671 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.543291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.1:38769 (local address 127.5.74.62:43199)
0923 23:09:23.543368 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.543371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.543675 (+ 304us) server_negotiation.cc:407] Connection header received
0923 23:09:23.544237 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.544238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.544259 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.544316 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:23.545191 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.545920 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.547095 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.547303 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.547400 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.548350 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:23.548352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:23.548353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:23.548358 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:23.548376 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:23.548385 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:23.548447 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.548550 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.548588 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 23:09:23.548911 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.543278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:43199 (local address 127.5.74.1:38769)
0923 23:09:23.543532 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.543571 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.544100 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.544376 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.544379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.544388 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:23.545027 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.545034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.546041 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.546043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.546986 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.546988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.547416 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.547417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.547552 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.548141 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:09:23.548142 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:23.548213 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:09:23.548249 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:23.548449 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:23.548456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:23.548460 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.548553 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.548662 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
I20260923 23:09:23.549425 25307 heartbeater.cc:347] Connected to a master server at 127.5.74.62:43199
I20260923 23:09:23.549526 25307 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:23.549716 25307 heartbeater.cc:511] Master 127.5.74.62:43199 requested a full tablet report, sending...
I20260923 23:09:23.550275 25119 ts_manager.cc:194] Registered new tserver with Master: a32aeb1888dd4e209ab7bb1ac6ebf8d7 (127.5.74.1:38859)
I20260923 23:09:23.550962 25119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.1:38769
W20260923 23:09:23.630784 25310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:23.631088 25310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:23.631115 25310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:23.632484 25310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:23.632542 25310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:23.632571 25310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.2
I20260923 23:09:23.633988 25310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.74.2
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.2
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:23.634204 25310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:23.634469 25310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:23.639864 25315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:23.639931 25316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:23.639891 25318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:23.640306 25310 server_base.cc:1034] running on GCE node
I20260923 23:09:23.640519 25310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:23.640769 25310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:23.641939 25310 hybrid_clock.cc:648] HybridClock initialized: now 1790204963641895 us; error 58 us; skew 500 ppm
I20260923 23:09:23.643368 25310 webserver.cc:460] Webserver started at http://127.5.74.2:44053/ using document root <none> and password file <none>
I20260923 23:09:23.643635 25310 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:23.643682 25310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:23.643826 25310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:23.645018 25310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/data/instance:
uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "3ee9531124513d96d8d006c8a692bfc0"
server_key_iv: "cdf563b1878facdf745a067d5e093ec7"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.645430 25310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance:
uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "3ee9531124513d96d8d006c8a692bfc0"
server_key_iv: "cdf563b1878facdf745a067d5e093ec7"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.646876 25310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:09:23.647784 25324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.648108 25310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:23.648288 25310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/wal
uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "3ee9531124513d96d8d006c8a692bfc0"
server_key_iv: "cdf563b1878facdf745a067d5e093ec7"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.648397 25310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:23.663115 25310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:23.663483 25310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:23.663648 25310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:23.663903 25310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:23.664400 25310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:23.664458 25310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.664520 25310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:23.664546 25310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.670605 25310 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.2:41059
I20260923 23:09:23.670652 25437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.2:41059 every 8 connection(s)
I20260923 23:09:23.670993 25310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/data/info.pb
I20260923 23:09:23.671758 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 25310
I20260923 23:09:23.671878 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/wal/instance
I20260923 23:09:23.672132 5416 external_mini_cluster.cc:1428] Setting key 14c3793b0e7b17bcf2fa2ce28cb895ea
I20260923 23:09:23.673270 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.74.3:0
--local_ip_for_outbound_sockets=127.5.74.3
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43199
--builtin_ntp_servers=127.5.74.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:09:23.677040 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.671828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:50449 (local address 127.5.74.62:43199)
0923 23:09:23.671915 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.671918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.672307 (+ 389us) server_negotiation.cc:407] Connection header received
0923 23:09:23.672965 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.672966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.672978 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.673026 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:23.673811 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.674330 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.675348 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.675644 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.675717 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.676694 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:23.676695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:23.676696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:23.676701 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:23.676719 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:23.676724 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:23.676784 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.676879 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.676913 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 23:09:23.677188 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.671880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:43199 (local address 127.5.74.2:50449)
0923 23:09:23.672114 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.672204 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.672837 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.673129 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.673133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.673144 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:23.673720 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.673725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.674437 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.674438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.675250 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.675251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.675750 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.675751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.675914 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.676544 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:09:23.676544 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:23.676574 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:09:23.676600 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:23.676792 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:23.676795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:23.676798 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.676859 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.677006 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260923 23:09:23.677754 25438 heartbeater.cc:347] Connected to a master server at 127.5.74.62:43199
I20260923 23:09:23.677875 25438 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:23.678095 25438 heartbeater.cc:511] Master 127.5.74.62:43199 requested a full tablet report, sending...
I20260923 23:09:23.678587 25119 ts_manager.cc:194] Registered new tserver with Master: 1a3afdd0529c4a84bcd2e13ce34e23d1 (127.5.74.2:41059)
I20260923 23:09:23.678997 25119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.2:50449
W20260923 23:09:23.758394 25441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:23.758648 25441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:23.758673 25441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:23.759950 25441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:23.760005 25441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:23.760035 25441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.74.3
I20260923 23:09:23.761536 25441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.74.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.74.3
--webserver_port=0
--tserver_master_addrs=127.5.74.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.74.3
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:23.761839 25441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:23.762178 25441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:23.767831 25449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:23.767832 25446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:23.768169 25447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:23.768144 25441 server_base.cc:1034] running on GCE node
I20260923 23:09:23.768492 25441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:23.768821 25441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:23.770031 25441 hybrid_clock.cc:648] HybridClock initialized: now 1790204963770021 us; error 40 us; skew 500 ppm
I20260923 23:09:23.771624 25441 webserver.cc:460] Webserver started at http://127.5.74.3:45551/ using document root <none> and password file <none>
I20260923 23:09:23.771938 25441 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:23.772020 25441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:23.772221 25441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:23.773344 25441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/data/instance:
uuid: "64cd914d7abf4e689d8add26499411b7"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "0977b6b7db46d7586c22ab983246b53b"
server_key_iv: "75db47a4af31436d40991a6448a7e819"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.773727 25441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance:
uuid: "64cd914d7abf4e689d8add26499411b7"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "0977b6b7db46d7586c22ab983246b53b"
server_key_iv: "75db47a4af31436d40991a6448a7e819"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.775209 25441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:23.776135 25455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.776396 25441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:23.776532 25441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/wal
uuid: "64cd914d7abf4e689d8add26499411b7"
format_stamp: "Formatted at 2026-09-23 23:09:23 on dist-test-slave-j4c5"
server_key: "0977b6b7db46d7586c22ab983246b53b"
server_key_iv: "75db47a4af31436d40991a6448a7e819"
server_key_version: "encryptionkey@0"
I20260923 23:09:23.776625 25441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:23.795476 25441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:23.795938 25441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:23.796113 25441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:23.796428 25441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:09:23.796773 25441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:09:23.796829 25441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.796875 25441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:09:23.796900 25441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:23.803193 25441 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.3:45139
I20260923 23:09:23.803272 25568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.3:45139 every 8 connection(s)
I20260923 23:09:23.803642 25441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/data/info.pb
I20260923 23:09:23.809397 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 25441
I20260923 23:09:23.809546 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/wal/instance
I20260923 23:09:23.809562 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.804593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.3:57837 (local address 127.5.74.62:43199)
0923 23:09:23.804670 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.804672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.804913 (+ 241us) server_negotiation.cc:407] Connection header received
0923 23:09:23.805456 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.805457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.805470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.805537 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:23.806271 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.806793 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.807949 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.808144 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.808277 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.809189 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:23.809191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:23.809191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:23.809196 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:23.809222 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:23.809227 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:23.809299 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.809442 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.809476 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 23:09:23.809770 5416 external_mini_cluster.cc:1428] Setting key 235d9c9df16cfd72460881b2186c9f11
I20260923 23:09:23.809788 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.804554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:43199 (local address 127.5.74.3:57837)
0923 23:09:23.804763 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.804814 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.805305 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.805614 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.805618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.805627 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:23.806165 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.806169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.806962 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.806962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.807842 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.807845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.808287 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.808289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.808480 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.809034 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:09:23.809035 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:23.809065 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:09:23.809090 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:23.809300 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:23.809306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:23.809309 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.809443 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.809552 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260923 23:09:23.810376 25569 heartbeater.cc:347] Connected to a master server at 127.5.74.62:43199
I20260923 23:09:23.810498 25569 heartbeater.cc:464] Registering TS with master...
I20260923 23:09:23.810693 25569 heartbeater.cc:511] Master 127.5.74.62:43199 requested a full tablet report, sending...
I20260923 23:09:23.811077 25119 ts_manager.cc:194] Registered new tserver with Master: 64cd914d7abf4e689d8add26499411b7 (127.5.74.3:45139)
I20260923 23:09:23.811476 25119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.74.3:57837
I20260923 23:09:23.821863 5416 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:09:23.826637 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.823369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.5.74.62:43199)
0923 23:09:23.823461 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.823463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.823559 (+ 96us) server_negotiation.cc:407] Connection header received
0923 23:09:23.823666 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.823667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.823682 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.823739 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:23.823953 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.824514 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.825360 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.825565 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.825642 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.826312 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:23.826313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:23.826313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:23.826318 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:23.826332 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:23.826343 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:23.826447 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.826516 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.826542 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 23:09:23.828593 25119 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:09:23.830181 25119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:09:23.837059 25580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.832512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.5.74.3:45139)
0923 23:09:23.833211 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.833215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.833224 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:09:23.833259 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.833261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.833276 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.833347 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.833536 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.834241 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.835850 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.836651 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.836731 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.836826 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.836948 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.836983 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":79,"threads_started":1}
I20260923 23:09:23.837069 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.832430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:45139 (local address 127.0.0.1:42150)
0923 23:09:23.832638 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.832680 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.832755 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.833412 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.833412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.833414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.833474 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.833476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.834336 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.834337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.835755 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.835757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.836739 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.836740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.836828 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.836843 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.836920 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.836981 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":37,"thread_start_us":75,"threads_started":1}
I20260923 23:09:23.837415 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.832495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.2:41059 (local address 127.0.0.1:58230)
0923 23:09:23.832863 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.832870 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.832911 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.833102 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.833105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.833110 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.833195 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.833197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.834547 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.834547 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.835784 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.835786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.837185 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.837186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.837243 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.837246 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.837302 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.837346 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":125,"thread_start_us":87,"threads_started":1}
I20260923 23:09:23.837657 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.832708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.5.74.2:41059)
0923 23:09:23.832894 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.832900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.832912 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:09:23.832992 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.832995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.833011 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.833085 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.833289 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.834471 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.835890 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.836651 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.836731 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.836826 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.837517 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.837562 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260923 23:09:23.837728 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.832441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:38859 (local address 127.0.0.1:44090)
0923 23:09:23.832721 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.832730 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.832770 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.833102 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.833105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.833108 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.833195 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.833197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.835474 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.835475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.836451 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.836453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.837508 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.837509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.837565 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.837568 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.837617 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.837657 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":6,"thread_start_us":92,"threads_started":1}
I20260923 23:09:23.838011 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.832693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.5.74.1:38859)
0923 23:09:23.832894 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.832900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.832912 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:09:23.832955 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.832957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.832974 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.833068 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.833289 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.834459 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.836543 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.837315 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.837375 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.837474 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.837894 (+ 420us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.837935 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":98,"threads_started":1}
I20260923 23:09:23.838119 25503 tablet_service.cc:1467] Processing CreateTablet for tablet fc860de5dfb84267a8abf4a7edaafc19 (DEFAULT_TABLE table=test-table [id=0f104afccc25421ea4371aed881e3a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:23.838526 25503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc860de5dfb84267a8abf4a7edaafc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:23.838467 25372 tablet_service.cc:1467] Processing CreateTablet for tablet fc860de5dfb84267a8abf4a7edaafc19 (DEFAULT_TABLE table=test-table [id=0f104afccc25421ea4371aed881e3a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:23.838651 25241 tablet_service.cc:1467] Processing CreateTablet for tablet fc860de5dfb84267a8abf4a7edaafc19 (DEFAULT_TABLE table=test-table [id=0f104afccc25421ea4371aed881e3a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:09:23.838786 25372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc860de5dfb84267a8abf4a7edaafc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:23.838982 25241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc860de5dfb84267a8abf4a7edaafc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:23.841643 25583 tablet_bootstrap.cc:492] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: Bootstrap starting.
I20260923 23:09:23.842139 25584 tablet_bootstrap.cc:492] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7: Bootstrap starting.
I20260923 23:09:23.842622 25583 tablet_bootstrap.cc:654] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:23.842793 25584 tablet_bootstrap.cc:654] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:23.842990 25583 log.cc:824] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:23.843107 25584 log.cc:824] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:23.843463 25585 tablet_bootstrap.cc:492] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: Bootstrap starting.
I20260923 23:09:23.843959 25584 tablet_bootstrap.cc:492] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7: No bootstrap required, opened a new log
I20260923 23:09:23.843959 25583 tablet_bootstrap.cc:492] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: No bootstrap required, opened a new log
I20260923 23:09:23.844033 25584 ts_tablet_manager.cc:1397] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:09:23.844033 25583 ts_tablet_manager.cc:1397] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:09:23.844682 25585 tablet_bootstrap.cc:654] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:23.845026 25585 log.cc:824] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:23.845824 25585 tablet_bootstrap.cc:492] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: No bootstrap required, opened a new log
I20260923 23:09:23.845920 25585 ts_tablet_manager.cc:1397] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:09:23.846701 25583 raft_consensus.cc:348] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } }
I20260923 23:09:23.846706 25584 raft_consensus.cc:348] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } }
I20260923 23:09:23.846887 25584 raft_consensus.cc:374] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:23.846889 25583 raft_consensus.cc:374] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:23.846925 25584 raft_consensus.cc:729] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32aeb1888dd4e209ab7bb1ac6ebf8d7, State: Initialized, Role: FOLLOWER
I20260923 23:09:23.847077 25584 consensus_queue.cc:260] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } }
I20260923 23:09:23.846925 25583 raft_consensus.cc:729] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64cd914d7abf4e689d8add26499411b7, State: Initialized, Role: FOLLOWER
I20260923 23:09:23.847328 25583 consensus_queue.cc:260] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } }
I20260923 23:09:23.847375 25584 ts_tablet_manager.cc:1428] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:09:23.847517 25307 heartbeater.cc:503] Master 127.5.74.62:43199 was elected leader, sending a full tablet report...
I20260923 23:09:23.847599 25583 ts_tablet_manager.cc:1428] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 23:09:23.847771 25569 heartbeater.cc:503] Master 127.5.74.62:43199 was elected leader, sending a full tablet report...
I20260923 23:09:23.848807 25585 raft_consensus.cc:348] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } }
I20260923 23:09:23.849017 25585 raft_consensus.cc:374] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:23.849069 25585 raft_consensus.cc:729] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3afdd0529c4a84bcd2e13ce34e23d1, State: Initialized, Role: FOLLOWER
I20260923 23:09:23.849200 25585 consensus_queue.cc:260] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } }
I20260923 23:09:23.849466 25585 ts_tablet_manager.cc:1428] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 23:09:23.849545 25438 heartbeater.cc:503] Master 127.5.74.62:43199 was elected leader, sending a full tablet report...
W20260923 23:09:23.921681 25439 tablet.cc:2369] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:09:23.941916 25591 raft_consensus.cc:482] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:09:23.942050 25591 raft_consensus.cc:504] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } }
I20260923 23:09:23.942559 25591 leader_election.cc:288] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64cd914d7abf4e689d8add26499411b7 (127.5.74.3:45139), a32aeb1888dd4e209ab7bb1ac6ebf8d7 (127.5.74.1:38859)
I20260923 23:09:23.945439 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.942756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.3:45139 (local address 127.5.74.2:34849)
0923 23:09:23.942816 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.942828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.942879 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.943080 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.943081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.943083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.943158 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.943159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.943806 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.943807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.944602 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.944605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.945168 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.945169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.945245 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.945248 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.945318 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.945366 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 23:09:23.945439 25580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.942897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:34849 (local address 127.5.74.3:45139)
0923 23:09:23.942947 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.942949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.942958 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:09:23.942995 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.942996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.943003 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.943075 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.943232 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.943714 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.944703 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.945077 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.945168 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.945179 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.945324 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.945363 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 23:09:23.945786 25523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc860de5dfb84267a8abf4a7edaafc19" candidate_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64cd914d7abf4e689d8add26499411b7" is_pre_election: true
I20260923 23:09:23.945878 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.942756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:38859 (local address 127.5.74.2:44167)
0923 23:09:23.943046 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.943056 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:23.943143 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:23.943319 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:23.943320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:23.943321 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:09:23.943450 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.943452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.944144 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.944145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.944988 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:23.944990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.945624 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.945626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:23.945697 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.945701 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:09:23.945758 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 23:09:23.945793 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":69,"thread_start_us":146,"threads_started":1}
I20260923 23:09:23.945930 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.942853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.74.2:44167 (local address 127.5.74.1:38859)
0923 23:09:23.942922 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.942924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.943126 (+ 202us) server_negotiation.cc:407] Connection header received
0923 23:09:23.943222 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.943223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.943235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.943317 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:09:23.943542 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.944052 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.945097 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.945532 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.945602 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.945614 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.945777 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.945821 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:09:23.945964 25523 raft_consensus.cc:2454] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a3afdd0529c4a84bcd2e13ce34e23d1 in term 0.
I20260923 23:09:23.946244 25325 leader_election.cc:302] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a3afdd0529c4a84bcd2e13ce34e23d1, 64cd914d7abf4e689d8add26499411b7; no voters:
I20260923 23:09:23.946257 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc860de5dfb84267a8abf4a7edaafc19" candidate_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" is_pre_election: true
I20260923 23:09:23.946408 25261 raft_consensus.cc:2454] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a3afdd0529c4a84bcd2e13ce34e23d1 in term 0.
I20260923 23:09:23.946424 25591 raft_consensus.cc:2781] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:09:23.946501 25591 raft_consensus.cc:482] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:09:23.946542 25591 raft_consensus.cc:3037] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:23.947329 25591 raft_consensus.cc:504] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } }
I20260923 23:09:23.947512 25591 leader_election.cc:288] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [CANDIDATE]: Term 1 election: Requested vote from peers 64cd914d7abf4e689d8add26499411b7 (127.5.74.3:45139), a32aeb1888dd4e209ab7bb1ac6ebf8d7 (127.5.74.1:38859)
I20260923 23:09:23.947731 25523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc860de5dfb84267a8abf4a7edaafc19" candidate_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64cd914d7abf4e689d8add26499411b7"
I20260923 23:09:23.947734 25261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc860de5dfb84267a8abf4a7edaafc19" candidate_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7"
I20260923 23:09:23.947820 25523 raft_consensus.cc:3037] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:23.947844 25261 raft_consensus.cc:3037] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:23.948751 25523 raft_consensus.cc:2454] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a3afdd0529c4a84bcd2e13ce34e23d1 in term 1.
I20260923 23:09:23.948884 25261 raft_consensus.cc:2454] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a3afdd0529c4a84bcd2e13ce34e23d1 in term 1.
I20260923 23:09:23.948940 25325 leader_election.cc:302] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a3afdd0529c4a84bcd2e13ce34e23d1, 64cd914d7abf4e689d8add26499411b7; no voters:
I20260923 23:09:23.949064 25591 raft_consensus.cc:2781] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:23.949242 25591 raft_consensus.cc:686] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 1 LEADER]: Becoming Leader. State: Replica: 1a3afdd0529c4a84bcd2e13ce34e23d1, State: Running, Role: LEADER
I20260923 23:09:23.949368 25591 consensus_queue.cc:237] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } }
I20260923 23:09:23.950278 25119 catalog_manager.cc:5495] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a3afdd0529c4a84bcd2e13ce34e23d1 (127.5.74.2). New cstate: current_term: 1 leader_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a3afdd0529c4a84bcd2e13ce34e23d1" member_type: VOTER last_known_addr { host: "127.5.74.2" port: 41059 } health_report { overall_health: HEALTHY } } }
I20260923 23:09:23.957377 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.954522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.5.74.2:41059)
0923 23:09:23.954583 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:23.954585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:23.954601 (+ 16us) server_negotiation.cc:407] Connection header received
0923 23:09:23.954643 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:23.954644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:23.954665 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:23.954731 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:23.954982 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.955511 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.956497 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:23.956689 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:23.956756 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:23.957000 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:23.957108 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:23.957174 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:23.957267 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 23:09:23.957288 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 23:09:23.959836 25261 raft_consensus.cc:1261] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7 [term 1 FOLLOWER]: Refusing update from remote peer 1a3afdd0529c4a84bcd2e13ce34e23d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:23.959836 25523 raft_consensus.cc:1261] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [term 1 FOLLOWER]: Refusing update from remote peer 1a3afdd0529c4a84bcd2e13ce34e23d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:23.960412 25593 consensus_queue.cc:1035] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32aeb1888dd4e209ab7bb1ac6ebf8d7" member_type: VOTER last_known_addr { host: "127.5.74.1" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:23.960616 25591 consensus_queue.cc:1035] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64cd914d7abf4e689d8add26499411b7" member_type: VOTER last_known_addr { host: "127.5.74.3" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:23.965116 25598 mvcc.cc:205] Tried to move back new op lower bound from 7332679532374335488 to 7332679532337127424. Current Snapshot: MvccSnapshot[applied={T|T < 7332679532374335488}]
I20260923 23:09:23.965934 25597 mvcc.cc:205] Tried to move back new op lower bound from 7332679532374335488 to 7332679532337127424. Current Snapshot: MvccSnapshot[applied={T|T < 7332679532374335488}]
I20260923 23:09:23.966377 25118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:23.966478 25118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:09:23.967308 25118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f728bf7e47f8452381e5eeee61d99588: Sending DeleteTablet for 3 replicas of tablet fc860de5dfb84267a8abf4a7edaafc19
I20260923 23:09:23.967679 25372 tablet_service.cc:1514] Processing DeleteTablet for tablet fc860de5dfb84267a8abf4a7edaafc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:23 UTC) from {username='slave'} at 127.0.0.1:58230
I20260923 23:09:23.967698 25241 tablet_service.cc:1514] Processing DeleteTablet for tablet fc860de5dfb84267a8abf4a7edaafc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:23 UTC) from {username='slave'} at 127.0.0.1:44090
I20260923 23:09:23.968047 25605 tablet_replica.cc:330] T fc860de5dfb84267a8abf4a7edaafc19 P a32aeb1888dd4e209ab7bb1ac6ebf8d7: stopping tablet replica
I20260923 23:09:23.968206 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 25179
I20260923 23:09:23.968302 25503 tablet_service.cc:1514] Processing DeleteTablet for tablet fc860de5dfb84267a8abf4a7edaafc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:09:23 UTC) from {username='slave'} at 127.0.0.1:42150
I20260923 23:09:23.969265 25607 tablet_replica.cc:330] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: stopping tablet replica
I20260923 23:09:23.969362 25607 raft_consensus.cc:2229] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:09:23.969470 25607 raft_consensus.cc:2258] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:23.969780 25606 tablet_replica.cc:330] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: stopping tablet replica
I20260923 23:09:23.969915 25606 raft_consensus.cc:2229] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:09:23.969919 25607 ts_tablet_manager.cc:1905] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:09:23.970036 25606 raft_consensus.cc:2258] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:09:23.970538 25606 ts_tablet_manager.cc:1905] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:09:23.971736 25607 ts_tablet_manager.cc:1918] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:23.971813 25607 log.cc:1196] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-2/wal/wals/fc860de5dfb84267a8abf4a7edaafc19
I20260923 23:09:23.972105 25607 ts_tablet_manager.cc:1939] T fc860de5dfb84267a8abf4a7edaafc19 P 64cd914d7abf4e689d8add26499411b7: Deleting consensus metadata
I20260923 23:09:23.972306 25606 ts_tablet_manager.cc:1918] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:09:23.972399 25606 log.cc:1196] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: Deleting WAL directory at /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204714633713-5416-0/minicluster-data/ts-1/wal/wals/fc860de5dfb84267a8abf4a7edaafc19
I20260923 23:09:23.972637 25606 ts_tablet_manager.cc:1939] T fc860de5dfb84267a8abf4a7edaafc19 P 1a3afdd0529c4a84bcd2e13ce34e23d1: Deleting consensus metadata
I20260923 23:09:23.972851 25103 catalog_manager.cc:4841] TS 64cd914d7abf4e689d8add26499411b7 (127.5.74.3:45139): tablet fc860de5dfb84267a8abf4a7edaafc19 (table test-table [id=0f104afccc25421ea4371aed881e3a70]) successfully deleted
I20260923 23:09:23.973081 25106 catalog_manager.cc:4841] TS 1a3afdd0529c4a84bcd2e13ce34e23d1 (127.5.74.2:41059): tablet fc860de5dfb84267a8abf4a7edaafc19 (table test-table [id=0f104afccc25421ea4371aed881e3a70]) successfully deleted
W20260923 23:09:23.974406 25104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.74.1:38859 (error 108)
I20260923 23:09:23.974609 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 25310
I20260923 23:09:23.974984 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:23.974866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.1:38859 (local address 127.0.0.1:44102)
0923 23:09:23.974913 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:23.974945 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.1:38859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 23:09:23.975502 25104 catalog_manager.cc:4584] TS a32aeb1888dd4e209ab7bb1ac6ebf8d7 (127.5.74.1:38859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc860de5dfb84267a8abf4a7edaafc19: Network error: Client connection negotiation failed: client connection to 127.5.74.1:38859: connect: Connection refused (error 111)
I20260923 23:09:23.980479 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 25441
I20260923 23:09:23.986157 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 25088
2026-09-23T23:09:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (788 ms)
[----------] 6 tests from EncryptionPolicyTest (8969 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:24 dist-test-slave-j4c5 krb5kdc[25611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:24 dist-test-slave-j4c5 krb5kdc[25611](info): set up 2 sockets
Sep 23 23:09:24 dist-test-slave-j4c5 krb5kdc[25611](info): commencing operation
krb5kdc: starting...
W20260923 23:09:25.052641 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:25 dist-test-slave-j4c5 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:25.184585 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:25.267638 25624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:25.267891 25624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:25.267917 25624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:25.269347 25624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:25.269433 25624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:25.269461 25624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:25.269476 25624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:25.269490 25624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:25.271468 25624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:25.271727 25624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:25.271927 25624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:25.277657 25632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:25.277719 25630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:25.277776 25629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:25.278084 25624 server_base.cc:1034] running on GCE node
I20260923 23:09:25.278381 25624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:25.278740 25624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:25.279953 25624 hybrid_clock.cc:648] HybridClock initialized: now 1790204965279945 us; error 57 us; skew 500 ppm
Sep 23 23:09:25 dist-test-slave-j4c5 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:25.282995 25624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:25.283759 25624 webserver.cc:460] Webserver started at http://127.0.0.1:45187/ using document root <none> and password file <none>
I20260923 23:09:25.284044 25624 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:25.284124 25624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:25.284343 25624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:25.285732 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "727463c445344bb791eeb31a2b1333dc"
format_stamp: "Formatted at 2026-09-23 23:09:25 on dist-test-slave-j4c5"
server_key: "28da1595f32678a19163bf14208bd440"
server_key_iv: "922675ecfd780e1d289b8f67950b9045"
server_key_version: "encryptionkey@0"
I20260923 23:09:25.286197 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "727463c445344bb791eeb31a2b1333dc"
format_stamp: "Formatted at 2026-09-23 23:09:25 on dist-test-slave-j4c5"
server_key: "28da1595f32678a19163bf14208bd440"
server_key_iv: "922675ecfd780e1d289b8f67950b9045"
server_key_version: "encryptionkey@0"
I20260923 23:09:25.287967 25624 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260923 23:09:25.289170 25639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:25.289500 25624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:25.289631 25624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "727463c445344bb791eeb31a2b1333dc"
format_stamp: "Formatted at 2026-09-23 23:09:25 on dist-test-slave-j4c5"
server_key: "28da1595f32678a19163bf14208bd440"
server_key_iv: "922675ecfd780e1d289b8f67950b9045"
server_key_version: "encryptionkey@0"
I20260923 23:09:25.289763 25624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:25.310225 25624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:25.313205 25624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:25.313414 25624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:25.317951 25624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35855
I20260923 23:09:25.318053 25691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35855 every 8 connection(s)
I20260923 23:09:25.318434 25624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:25.319749 25692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:25.320402 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 25624
I20260923 23:09:25.320542 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:25.320748 5416 external_mini_cluster.cc:1428] Setting key 02f03fbfd90c528bbb49953e0aa1fe6a
I20260923 23:09:25.323485 25692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc: Bootstrap starting.
I20260923 23:09:25.324326 25692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:25.324681 25692 log.cc:824] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:25.325592 25692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc: No bootstrap required, opened a new log
I20260923 23:09:25.328431 25692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727463c445344bb791eeb31a2b1333dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35855 } }
I20260923 23:09:25.328635 25692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:25.328677 25692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727463c445344bb791eeb31a2b1333dc, State: Initialized, Role: FOLLOWER
I20260923 23:09:25.328851 25692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727463c445344bb791eeb31a2b1333dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35855 } }
I20260923 23:09:25.328951 25692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:25.328996 25692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:25.329052 25692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [term 0 FOLLOWER]: Advancing to term 1
Sep 23 23:09:25 dist-test-slave-j4c5 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 23:09:25.330041 25692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727463c445344bb791eeb31a2b1333dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35855 } }
I20260923 23:09:25.330173 25692 leader_election.cc:302] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 727463c445344bb791eeb31a2b1333dc; no voters:
I20260923 23:09:25.330443 25692 leader_election.cc:288] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:25.330518 25697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:25.330717 25697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [term 1 LEADER]: Becoming Leader. State: Replica: 727463c445344bb791eeb31a2b1333dc, State: Running, Role: LEADER
I20260923 23:09:25.330797 25692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:25.330824 25697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727463c445344bb791eeb31a2b1333dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35855 } }
I20260923 23:09:25.331281 25699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "727463c445344bb791eeb31a2b1333dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727463c445344bb791eeb31a2b1333dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35855 } } }
I20260923 23:09:25.331271 25697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 727463c445344bb791eeb31a2b1333dc. Latest consensus state: current_term: 1 leader_uuid: "727463c445344bb791eeb31a2b1333dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727463c445344bb791eeb31a2b1333dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35855 } } }
I20260923 23:09:25.331455 25697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:25.331447 25699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc [sys.catalog]: This master's current role is: LEADER
W20260923 23:09:25.333076 25713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:25.333139 25713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:25.333186 25706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:25.333359 25706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:25.335067 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:25.321420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56579 (local address 127.0.0.1:35855)
0923 23:09:25.321676 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:25.321688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:25.321728 (+ 40us) server_negotiation.cc:407] Connection header received
0923 23:09:25.322343 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:25.322353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:25.322424 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:25.322611 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:25.323303 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:25.324898 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:25.326076 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:25.326831 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:25.326975 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:25.329956 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:25.329962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:25.329962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:25.329977 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:25.332790 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:25.333405 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:25.333408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:25.333409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:25.333522 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:25.333807 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:25.333808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:25.333808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:25.334290 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:25.334572 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:25.334756 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 23:09:25.334875 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":130,"threads_started":1}
I20260923 23:09:25.335403 25706 catalog_manager.cc:1347] Generated new cluster ID: 60c676e72cce4d278867bb3fd6115c05
I20260923 23:09:25.335448 25706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:25.336189 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:25.343364 25706 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:25.344199 25706 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:25.349566 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:25.337626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48605 (local address 127.0.0.1:35855)
0923 23:09:25.342793 (+ 5167us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:25.342795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:25.342812 (+ 17us) server_negotiation.cc:407] Connection header received
0923 23:09:25.342839 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:25.342840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:25.342864 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:25.342960 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:25.343770 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:25.344426 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:25.345152 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:25.345397 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:25.345473 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:25.346774 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:25.346776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:25.346776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:25.346784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:25.348329 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:25.348716 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:25.348717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:25.348718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:25.348783 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:25.349026 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:25.349027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:25.349027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:25.349149 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:25.349273 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:25.349398 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 23:09:25.349464 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5129}
I20260923 23:09:25.360621 25706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 727463c445344bb791eeb31a2b1333dc: Generated new TSK 0
I20260923 23:09:25.360878 25706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:25.367254 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 25624
W20260923 23:09:25.371797 25609 connection.cc:535] client connection to 127.0.0.1:35855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35855): Connection reset by peer (error 104)
2026-09-23T23:09:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:25 dist-test-slave-j4c5 krb5kdc[25723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:25 dist-test-slave-j4c5 krb5kdc[25723](info): set up 2 sockets
Sep 23 23:09:25 dist-test-slave-j4c5 krb5kdc[25723](info): commencing operation
krb5kdc: starting...
W20260923 23:09:26.452983 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:26 dist-test-slave-j4c5 krb5kdc[25723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:26.588845 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:26.676112 25736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:26.676373 25736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:26.676397 25736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:26.677551 25736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:26.677591 25736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:26.677606 25736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:26.677618 25736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:26.677630 25736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:26.679087 25736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:26.679308 25736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:26.679486 25736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:26.685000 25742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:26.685077 25744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:26.685022 25741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:26.685041 25736 server_base.cc:1034] running on GCE node
I20260923 23:09:26.685599 25736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:26.685964 25736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:26.687189 25736 hybrid_clock.cc:648] HybridClock initialized: now 1790204966687166 us; error 55 us; skew 500 ppm
Sep 23 23:09:26 dist-test-slave-j4c5 krb5kdc[25723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:26.697680 25736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:26.698278 25736 webserver.cc:460] Webserver started at http://127.0.0.1:37189/ using document root <none> and password file <none>
I20260923 23:09:26.698560 25736 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:26.698637 25736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:26.698802 25736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:26.699965 25736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "5d9578fa3e8842c197444b4617f9f9aa"
format_stamp: "Formatted at 2026-09-23 23:09:26 on dist-test-slave-j4c5"
server_key: "5eaaedec4921d40d6bad655214d8815a"
server_key_iv: "4b6dbe483e319c598b8e16af225f3918"
server_key_version: "encryptionkey@0"
I20260923 23:09:26.700393 25736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "5d9578fa3e8842c197444b4617f9f9aa"
format_stamp: "Formatted at 2026-09-23 23:09:26 on dist-test-slave-j4c5"
server_key: "5eaaedec4921d40d6bad655214d8815a"
server_key_iv: "4b6dbe483e319c598b8e16af225f3918"
server_key_version: "encryptionkey@0"
I20260923 23:09:26.701824 25736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:26.702602 25751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:26.702814 25736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:26.702930 25736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "5d9578fa3e8842c197444b4617f9f9aa"
format_stamp: "Formatted at 2026-09-23 23:09:26 on dist-test-slave-j4c5"
server_key: "5eaaedec4921d40d6bad655214d8815a"
server_key_iv: "4b6dbe483e319c598b8e16af225f3918"
server_key_version: "encryptionkey@0"
I20260923 23:09:26.703014 25736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:26.718993 25736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:26.721508 25736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:26.721709 25736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:26.725976 25736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35665
I20260923 23:09:26.726012 25803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35665 every 8 connection(s)
I20260923 23:09:26.726423 25736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:26.727434 25804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:26.729801 25804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa: Bootstrap starting.
I20260923 23:09:26.730350 25804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:26.730593 25804 log.cc:824] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:26.731444 25804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa: No bootstrap required, opened a new log
I20260923 23:09:26.733358 25804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9578fa3e8842c197444b4617f9f9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260923 23:09:26.733526 25804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:26.733552 25804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9578fa3e8842c197444b4617f9f9aa, State: Initialized, Role: FOLLOWER
I20260923 23:09:26.733662 25804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9578fa3e8842c197444b4617f9f9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260923 23:09:26.733719 25804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:26.733741 25804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:26.733806 25804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:26.734409 25804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9578fa3e8842c197444b4617f9f9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260923 23:09:26.734503 25804 leader_election.cc:302] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d9578fa3e8842c197444b4617f9f9aa; no voters:
I20260923 23:09:26.734709 25804 leader_election.cc:288] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:26.734774 25807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:26.735003 25804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:26.735045 25807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [term 1 LEADER]: Becoming Leader. State: Replica: 5d9578fa3e8842c197444b4617f9f9aa, State: Running, Role: LEADER
I20260923 23:09:26.735131 25807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9578fa3e8842c197444b4617f9f9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260923 23:09:26.735343 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 25736
I20260923 23:09:26.735491 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:26.735702 5416 external_mini_cluster.cc:1428] Setting key 7480c7c6630bfe2741874f783ef2ab70
I20260923 23:09:26.736209 25808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d9578fa3e8842c197444b4617f9f9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9578fa3e8842c197444b4617f9f9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260923 23:09:26.736312 25808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:26.736464 25809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9578fa3e8842c197444b4617f9f9aa. Latest consensus state: current_term: 1 leader_uuid: "5d9578fa3e8842c197444b4617f9f9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9578fa3e8842c197444b4617f9f9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260923 23:09:26.736537 25809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:26.736640 25820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:26.737236 25820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:26.738955 25820 catalog_manager.cc:1347] Generated new cluster ID: b4bb1fb45aaf4a408551cc596859839d
I20260923 23:09:26.739014 25820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:26.741955 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:26.736393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39071 (local address 127.0.0.1:35665)
0923 23:09:26.736611 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:26.736621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:26.736790 (+ 169us) server_negotiation.cc:407] Connection header received
0923 23:09:26.737083 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:26.737090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:26.737139 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:26.737310 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:26.737568 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:26.738883 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:26.739790 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:26.740353 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:26.740495 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:26.741290 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:26.741294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:26.741294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:26.741499 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:26.741531 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:26.741543 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:26.741644 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:26.741731 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 23:09:26.741764 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260923 23:09:26.745357 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:26.755717 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:26.752587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41927 (local address 127.0.0.1:35665)
0923 23:09:26.752644 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:26.752647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:26.752809 (+ 162us) server_negotiation.cc:407] Connection header received
0923 23:09:26.752863 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:26.752864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:26.752890 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:26.752961 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:26.753134 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:26.753654 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:26.754397 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:26.754626 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:26.754704 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:26.755428 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:26.755429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:26.755430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:26.755435 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:26.755451 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:26.755456 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:26.755533 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:26.755605 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 23:09:26.755619 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:09:26.771055 25820 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:26.771741 25820 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:26.776769 25820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d9578fa3e8842c197444b4617f9f9aa: Generated new TSK 0
I20260923 23:09:26.776981 25820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:26.784015 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 25736
W20260923 23:09:26.788702 25721 connection.cc:535] client connection to 127.0.0.1:35665 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35665): Connection reset by peer (error 104)
2026-09-23T23:09:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:26 dist-test-slave-j4c5 krb5kdc[25835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:26 dist-test-slave-j4c5 krb5kdc[25835](info): set up 2 sockets
Sep 23 23:09:26 dist-test-slave-j4c5 krb5kdc[25835](info): commencing operation
krb5kdc: starting...
W20260923 23:09:27.872344 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:27 dist-test-slave-j4c5 krb5kdc[25835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:28.009354 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:28.098699 25848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:28.099036 25848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:28.099094 25848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:28.100739 25848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:28.100848 25848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:28.100891 25848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:28.100934 25848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:28.100965 25848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:28.103165 25848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:28.103510 25848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:28.103763 25848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:28.109860 25853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:28.109849 25856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:28.109849 25854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:28.110787 25848 server_base.cc:1034] running on GCE node
I20260923 23:09:28.111171 25848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:28.111562 25848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:28.112780 25848 hybrid_clock.cc:648] HybridClock initialized: now 1790204968112763 us; error 70 us; skew 500 ppm
Sep 23 23:09:28 dist-test-slave-j4c5 krb5kdc[25835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:28.115693 25848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:28.116397 25848 webserver.cc:460] Webserver started at http://127.0.0.1:43447/ using document root <none> and password file <none>
I20260923 23:09:28.116696 25848 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:28.116748 25848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:28.116919 25848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:28.118088 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "b70d232858264f99b1b326c412fd8dfd"
format_stamp: "Formatted at 2026-09-23 23:09:28 on dist-test-slave-j4c5"
server_key: "0165689ab3708579d19d6c780e3ea9f4"
server_key_iv: "d4e61905c9c68134c899f97a1149bd06"
server_key_version: "encryptionkey@0"
I20260923 23:09:28.118516 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "b70d232858264f99b1b326c412fd8dfd"
format_stamp: "Formatted at 2026-09-23 23:09:28 on dist-test-slave-j4c5"
server_key: "0165689ab3708579d19d6c780e3ea9f4"
server_key_iv: "d4e61905c9c68134c899f97a1149bd06"
server_key_version: "encryptionkey@0"
I20260923 23:09:28.120115 25848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:09:28.121111 25863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:28.121374 25848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:28.121492 25848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "b70d232858264f99b1b326c412fd8dfd"
format_stamp: "Formatted at 2026-09-23 23:09:28 on dist-test-slave-j4c5"
server_key: "0165689ab3708579d19d6c780e3ea9f4"
server_key_iv: "d4e61905c9c68134c899f97a1149bd06"
server_key_version: "encryptionkey@0"
I20260923 23:09:28.121582 25848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:28.125515 25848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:28.127877 25848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:28.128037 25848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:28.132193 25848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43061
I20260923 23:09:28.132269 25915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43061 every 8 connection(s)
I20260923 23:09:28.132705 25848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:28.133764 25916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:28.135159 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 25848
I20260923 23:09:28.135299 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:28.135493 5416 external_mini_cluster.cc:1428] Setting key 2b4f42b0995aaf53fbb74652241483de
I20260923 23:09:28.136732 25916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd: Bootstrap starting.
I20260923 23:09:28.137493 25916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:28.137866 25916 log.cc:824] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:28.138170 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:28.136080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53199 (local address 127.0.0.1:43061)
0923 23:09:28.136362 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:28.136372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:28.136401 (+ 29us) server_negotiation.cc:407] Connection header received
0923 23:09:28.136829 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:28.136838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:28.136878 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:28.137031 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:28.137708 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:28.137712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:28.137712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:28.137716 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:28.137771 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:28.137782 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:28.137841 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:28.137959 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 23:09:28.138005 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":175,"threads_started":1}
I20260923 23:09:28.138616 25916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd: No bootstrap required, opened a new log
I20260923 23:09:28.139256 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:28.141209 25916 raft_consensus.cc:348] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d232858264f99b1b326c412fd8dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43061 } }
I20260923 23:09:28.141390 25916 raft_consensus.cc:374] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:28.141438 25916 raft_consensus.cc:729] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70d232858264f99b1b326c412fd8dfd, State: Initialized, Role: FOLLOWER
I20260923 23:09:28.141589 25916 consensus_queue.cc:260] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d232858264f99b1b326c412fd8dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43061 } }
I20260923 23:09:28.141686 25916 raft_consensus.cc:388] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:28.141728 25916 raft_consensus.cc:482] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:28.141780 25916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:28.142076 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:28.140955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44179 (local address 127.0.0.1:43061)
0923 23:09:28.141030 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:28.141032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:28.141129 (+ 97us) server_negotiation.cc:407] Connection header received
0923 23:09:28.141198 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:28.141199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:28.141204 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:28.141250 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:28.141879 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:28.141881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:28.141881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:28.141886 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:28.141902 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:28.141906 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:28.141975 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:28.142010 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 23:09:28.142023 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 23:09:28.142539 25916 raft_consensus.cc:504] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d232858264f99b1b326c412fd8dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43061 } }
I20260923 23:09:28.142668 25916 leader_election.cc:302] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b70d232858264f99b1b326c412fd8dfd; no voters:
I20260923 23:09:28.142859 25916 leader_election.cc:288] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:28.142891 25926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:28.143052 25926 raft_consensus.cc:686] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [term 1 LEADER]: Becoming Leader. State: Replica: b70d232858264f99b1b326c412fd8dfd, State: Running, Role: LEADER
I20260923 23:09:28.143139 25916 sys_catalog.cc:564] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:28.143194 25926 consensus_queue.cc:237] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d232858264f99b1b326c412fd8dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43061 } }
I20260923 23:09:28.143632 25927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b70d232858264f99b1b326c412fd8dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d232858264f99b1b326c412fd8dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43061 } } }
I20260923 23:09:28.143642 25928 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b70d232858264f99b1b326c412fd8dfd. Latest consensus state: current_term: 1 leader_uuid: "b70d232858264f99b1b326c412fd8dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d232858264f99b1b326c412fd8dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43061 } } }
I20260923 23:09:28.143771 25928 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:28.143930 25927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:28.144222 25935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:28.144641 25935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:28.146509 25935 catalog_manager.cc:1347] Generated new cluster ID: 5af79270836c457699427d2e34e94883
I20260923 23:09:28.146571 25935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:28.154660 25935 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:28.154863 25935 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:28.158903 25935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b70d232858264f99b1b326c412fd8dfd: Generated new TSK 0
I20260923 23:09:28.159147 25935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:28.161504 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 25848
2026-09-23T23:09:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:28 dist-test-slave-j4c5 krb5kdc[25947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:28 dist-test-slave-j4c5 krb5kdc[25947](info): set up 2 sockets
Sep 23 23:09:28 dist-test-slave-j4c5 krb5kdc[25947](info): commencing operation
krb5kdc: starting...
W20260923 23:09:29.233211 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:29 dist-test-slave-j4c5 krb5kdc[25947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:29.371600 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:29.458689 25960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:29.458937 25960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:29.458962 25960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:29.460330 25960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:29.460381 25960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:29.460394 25960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:29.460407 25960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:29.460419 25960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:29.462177 25960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:29.462432 25960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:29.462618 25960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:29.467986 25968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:29.467958 25966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:29.467958 25965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:29.468254 25960 server_base.cc:1034] running on GCE node
I20260923 23:09:29.468569 25960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:29.468899 25960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:29.470103 25960 hybrid_clock.cc:648] HybridClock initialized: now 1790204969470082 us; error 57 us; skew 500 ppm
Sep 23 23:09:29 dist-test-slave-j4c5 krb5kdc[25947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:29.473403 25960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:29.473879 25960 webserver.cc:460] Webserver started at http://127.0.0.1:40857/ using document root <none> and password file <none>
I20260923 23:09:29.474133 25960 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:29.474206 25960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:29.474361 25960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:29.475510 25960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "a0118ae56acf4706ae5226577dfde09d"
format_stamp: "Formatted at 2026-09-23 23:09:29 on dist-test-slave-j4c5"
server_key: "d5453a9979eef4ff4aa8748285818d01"
server_key_iv: "a64ff426f1ebde7befddcd1885607a59"
server_key_version: "encryptionkey@0"
I20260923 23:09:29.475924 25960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "a0118ae56acf4706ae5226577dfde09d"
format_stamp: "Formatted at 2026-09-23 23:09:29 on dist-test-slave-j4c5"
server_key: "d5453a9979eef4ff4aa8748285818d01"
server_key_iv: "a64ff426f1ebde7befddcd1885607a59"
server_key_version: "encryptionkey@0"
I20260923 23:09:29.477545 25960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:29.478391 25975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:29.478600 25960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:29.478710 25960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "a0118ae56acf4706ae5226577dfde09d"
format_stamp: "Formatted at 2026-09-23 23:09:29 on dist-test-slave-j4c5"
server_key: "d5453a9979eef4ff4aa8748285818d01"
server_key_iv: "a64ff426f1ebde7befddcd1885607a59"
server_key_version: "encryptionkey@0"
I20260923 23:09:29.478794 25960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:29.496793 25960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:29.500033 25960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:29.500257 25960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:29.504601 25960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36787
I20260923 23:09:29.504753 26027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36787 every 8 connection(s)
I20260923 23:09:29.505060 25960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:29.506229 26028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:29.507691 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 25960
I20260923 23:09:29.507812 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:29.508037 5416 external_mini_cluster.cc:1428] Setting key ff6f10b353c4ded560825ea8afaba72b
I20260923 23:09:29.509358 26028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d: Bootstrap starting.
I20260923 23:09:29.510061 26028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:29.510393 26028 log.cc:824] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:29.511284 26028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d: No bootstrap required, opened a new log
I20260923 23:09:29.513795 26028 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0118ae56acf4706ae5226577dfde09d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36787 } }
I20260923 23:09:29.513989 26028 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:29.514032 26028 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0118ae56acf4706ae5226577dfde09d, State: Initialized, Role: FOLLOWER
I20260923 23:09:29.514223 26028 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0118ae56acf4706ae5226577dfde09d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36787 } }
I20260923 23:09:29.514328 26028 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:29.514372 26028 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:29.514426 26028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:29.515246 26028 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0118ae56acf4706ae5226577dfde09d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36787 } }
I20260923 23:09:29.515372 26028 leader_election.cc:302] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0118ae56acf4706ae5226577dfde09d; no voters:
I20260923 23:09:29.515591 26028 leader_election.cc:288] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 23 23:09:29 dist-test-slave-j4c5 krb5kdc[25947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 23:09:29.515690 26033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:29.515928 26033 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [term 1 LEADER]: Becoming Leader. State: Replica: a0118ae56acf4706ae5226577dfde09d, State: Running, Role: LEADER
I20260923 23:09:29.515962 26028 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:29.517551 26033 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0118ae56acf4706ae5226577dfde09d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36787 } }
I20260923 23:09:29.518291 26034 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0118ae56acf4706ae5226577dfde09d. Latest consensus state: current_term: 1 leader_uuid: "a0118ae56acf4706ae5226577dfde09d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0118ae56acf4706ae5226577dfde09d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36787 } } }
I20260923 23:09:29.518272 26035 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0118ae56acf4706ae5226577dfde09d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0118ae56acf4706ae5226577dfde09d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36787 } } }
I20260923 23:09:29.518393 26034 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:29.518419 26035 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d [sys.catalog]: This master's current role is: LEADER
W20260923 23:09:29.519073 26047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:29.519145 26047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:29.519234 26048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:29.519414 26048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:29.520819 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:29.508732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53227 (local address 127.0.0.1:36787)
0923 23:09:29.508957 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:29.508975 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:29.509012 (+ 37us) server_negotiation.cc:407] Connection header received
0923 23:09:29.509468 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:29.509474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:29.509573 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:29.509741 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:29.510515 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:29.511706 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:29.512616 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:29.513066 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:29.513211 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:29.516320 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:29.516325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:29.516325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:29.516338 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:29.518899 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:29.519387 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:29.519390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:29.519391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:29.519502 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:29.519726 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:29.519727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:29.519728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:29.520089 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:29.520264 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:29.520463 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 23:09:29.520614 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
I20260923 23:09:29.521117 26048 catalog_manager.cc:1347] Generated new cluster ID: 70d177b6e09244ca875036d32bc7f6ad
I20260923 23:09:29.521168 26048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:29.521752 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:29.531306 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:29.523235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55671 (local address 127.0.0.1:36787)
0923 23:09:29.523271 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:29.523273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:29.523301 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:09:29.523357 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:29.523358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:29.523368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:29.523439 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:29.524130 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:29.524988 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:29.526028 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:29.526377 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:29.526457 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:29.527917 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:29.527919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:29.527919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:29.527927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:29.529800 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:29.530245 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:29.530246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:29.530247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:29.530311 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:29.530530 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:29.530530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:29.530531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:29.530673 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:29.530800 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:29.531105 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 23:09:29.531195 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:09:29.537510 26048 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:29.538290 26048 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:29.547446 26048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0118ae56acf4706ae5226577dfde09d: Generated new TSK 0
I20260923 23:09:29.547650 26048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:29.553350 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 25960
2026-09-23T23:09:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:29 dist-test-slave-j4c5 krb5kdc[26059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:29 dist-test-slave-j4c5 krb5kdc[26059](info): set up 2 sockets
Sep 23 23:09:29 dist-test-slave-j4c5 krb5kdc[26059](info): commencing operation
krb5kdc: starting...
W20260923 23:09:30.608512 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:30 dist-test-slave-j4c5 krb5kdc[26059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:30.744479 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:30.830001 26072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:30.830318 26072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:30.830365 26072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:30.831722 26072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:30.831813 26072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:30.831836 26072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:30.831857 26072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:30.831887 26072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:30.833776 26072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:30.834137 26072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:30.834425 26072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:09:30.841296 26072 server_base.cc:1034] running on GCE node
W20260923 23:09:30.841264 26077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:30.841264 26080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:30.841266 26078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:30.841828 26072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:30.842155 26072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:30.843354 26072 hybrid_clock.cc:648] HybridClock initialized: now 1790204970843318 us; error 59 us; skew 500 ppm
Sep 23 23:09:30 dist-test-slave-j4c5 krb5kdc[26059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:30.846312 26072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:30.846885 26072 webserver.cc:460] Webserver started at http://127.0.0.1:35823/ using document root <none> and password file <none>
I20260923 23:09:30.847139 26072 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:30.847213 26072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:30.847370 26072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:30.848588 26072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3"
format_stamp: "Formatted at 2026-09-23 23:09:30 on dist-test-slave-j4c5"
server_key: "7f5c2baace629962547cc05b63d526f6"
server_key_iv: "2613afdb8407eeba844f1125eeec100e"
server_key_version: "encryptionkey@0"
I20260923 23:09:30.848986 26072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3"
format_stamp: "Formatted at 2026-09-23 23:09:30 on dist-test-slave-j4c5"
server_key: "7f5c2baace629962547cc05b63d526f6"
server_key_iv: "2613afdb8407eeba844f1125eeec100e"
server_key_version: "encryptionkey@0"
I20260923 23:09:30.850664 26072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:30.851603 26087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:30.851892 26072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:30.852061 26072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3"
format_stamp: "Formatted at 2026-09-23 23:09:30 on dist-test-slave-j4c5"
server_key: "7f5c2baace629962547cc05b63d526f6"
server_key_iv: "2613afdb8407eeba844f1125eeec100e"
server_key_version: "encryptionkey@0"
I20260923 23:09:30.852164 26072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:30.863883 26072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:30.866601 26072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:30.866801 26072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:30.871465 26072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37947
I20260923 23:09:30.871524 26139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37947 every 8 connection(s)
I20260923 23:09:30.871980 26072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:30.873097 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:30.875512 26140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3: Bootstrap starting.
I20260923 23:09:30.876139 26140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:30.876430 26140 log.cc:824] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:30.877190 26140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3: No bootstrap required, opened a new log
I20260923 23:09:30.878806 26140 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260923 23:09:30.878983 26140 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:30.879009 26140 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c70a8dd3e240feb2cc9f3947b8fbd3, State: Initialized, Role: FOLLOWER
I20260923 23:09:30.879174 26140 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260923 23:09:30.879264 26140 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:30.879289 26140 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:30.879321 26140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:30.879977 26140 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260923 23:09:30.880026 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 26072
I20260923 23:09:30.880134 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:30.880125 26140 leader_election.cc:302] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5c70a8dd3e240feb2cc9f3947b8fbd3; no voters:
I20260923 23:09:30.880367 26140 leader_election.cc:288] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:30.880398 5416 external_mini_cluster.cc:1428] Setting key 55760180e448b3487e56ea7149ff0cdc
I20260923 23:09:30.880475 26143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:30.880678 26140 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:30.880707 26143 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [term 1 LEADER]: Becoming Leader. State: Replica: b5c70a8dd3e240feb2cc9f3947b8fbd3, State: Running, Role: LEADER
I20260923 23:09:30.880952 26143 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260923 23:09:30.881397 26145 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5c70a8dd3e240feb2cc9f3947b8fbd3. Latest consensus state: current_term: 1 leader_uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } } }
I20260923 23:09:30.881402 26144 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c70a8dd3e240feb2cc9f3947b8fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } } }
I20260923 23:09:30.881516 26145 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:30.881547 26144 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:30.881752 26153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:30.882403 26153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:30.884471 26153 catalog_manager.cc:1347] Generated new cluster ID: c0cc9f0d13cd4182ab109c8dce2ca015
I20260923 23:09:30.884544 26153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:30.887709 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:30.881010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59105 (local address 127.0.0.1:37947)
0923 23:09:30.881785 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:30.881799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:30.881841 (+ 42us) server_negotiation.cc:407] Connection header received
0923 23:09:30.881979 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:30.881987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:30.882059 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:30.882724 (+ 665us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:30.882823 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:30.884272 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:30.885458 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:30.886108 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:30.886228 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:30.886974 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:30.886979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:30.886979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:30.887199 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:30.887246 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:30.887262 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:30.887335 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:30.887408 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 23:09:30.887475 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":77,"threads_started":1}
I20260923 23:09:30.888701 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:30.894305 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:30.890300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59233 (local address 127.0.0.1:37947)
0923 23:09:30.890347 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:30.890349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:30.890498 (+ 149us) server_negotiation.cc:407] Connection header received
0923 23:09:30.890611 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:30.890612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:30.890624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:30.890655 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:30.890899 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:30.891663 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:30.892803 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:30.893112 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:30.893156 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:30.894008 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:30.894010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:30.894010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:30.894016 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:30.894034 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:30.894039 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:30.894068 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:30.894177 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 23:09:30.894209 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:09:30.920573 26153 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:30.921224 26153 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:30.933683 26153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5c70a8dd3e240feb2cc9f3947b8fbd3: Generated new TSK 0
I20260923 23:09:30.933964 26153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:30.936942 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 26072
2026-09-23T23:09:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:30 dist-test-slave-j4c5 krb5kdc[26171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:30 dist-test-slave-j4c5 krb5kdc[26171](info): set up 2 sockets
Sep 23 23:09:30 dist-test-slave-j4c5 krb5kdc[26171](info): commencing operation
krb5kdc: starting...
W20260923 23:09:32.016449 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:32 dist-test-slave-j4c5 krb5kdc[26171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:32.165594 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:32.253719 26184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:32.254002 26184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:32.254046 26184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:32.255378 26184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:32.255455 26184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:32.255479 26184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:32.255507 26184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:32.255548 26184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:32.257282 26184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:32.257619 26184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:32.257879 26184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:09:32.263743 26184 server_base.cc:1034] running on GCE node
W20260923 23:09:32.263715 26189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:32.263701 26192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:32.263722 26190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:32.264315 26184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:32.264678 26184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:32.265877 26184 hybrid_clock.cc:648] HybridClock initialized: now 1790204972265863 us; error 40 us; skew 500 ppm
Sep 23 23:09:32 dist-test-slave-j4c5 krb5kdc[26171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:32.268853 26184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:32.269348 26184 webserver.cc:460] Webserver started at http://127.0.0.1:33175/ using document root <none> and password file <none>
I20260923 23:09:32.269623 26184 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:32.269701 26184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:32.269856 26184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:32.271006 26184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "d718c03482fd43918a2868abf0f8a6c3"
format_stamp: "Formatted at 2026-09-23 23:09:32 on dist-test-slave-j4c5"
server_key: "2a9486ec852638532783b47fb2464d33"
server_key_iv: "2b9bc33569eb3b8df912ad3d8c151c4a"
server_key_version: "encryptionkey@0"
I20260923 23:09:32.271390 26184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "d718c03482fd43918a2868abf0f8a6c3"
format_stamp: "Formatted at 2026-09-23 23:09:32 on dist-test-slave-j4c5"
server_key: "2a9486ec852638532783b47fb2464d33"
server_key_iv: "2b9bc33569eb3b8df912ad3d8c151c4a"
server_key_version: "encryptionkey@0"
I20260923 23:09:32.273293 26184 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 23:09:32.274276 26199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:32.274535 26184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:32.274658 26184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "d718c03482fd43918a2868abf0f8a6c3"
format_stamp: "Formatted at 2026-09-23 23:09:32 on dist-test-slave-j4c5"
server_key: "2a9486ec852638532783b47fb2464d33"
server_key_iv: "2b9bc33569eb3b8df912ad3d8c151c4a"
server_key_version: "encryptionkey@0"
I20260923 23:09:32.274746 26184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:32.278541 26184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:32.281095 26184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:32.281275 26184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:32.285566 26184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35871
I20260923 23:09:32.285601 26251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35871 every 8 connection(s)
I20260923 23:09:32.285975 26184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:32.287156 26252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:32.289927 26252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3: Bootstrap starting.
I20260923 23:09:32.290818 26252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:32.291148 26252 log.cc:824] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:32.291970 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 26184
I20260923 23:09:32.292064 26252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3: No bootstrap required, opened a new log
I20260923 23:09:32.292136 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:32.292335 5416 external_mini_cluster.cc:1428] Setting key 00beacc6af0c12790da99e55986c6719
I20260923 23:09:32.294425 26252 raft_consensus.cc:348] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718c03482fd43918a2868abf0f8a6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } }
I20260923 23:09:32.294629 26252 raft_consensus.cc:374] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:32.294668 26252 raft_consensus.cc:729] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d718c03482fd43918a2868abf0f8a6c3, State: Initialized, Role: FOLLOWER
I20260923 23:09:32.294836 26252 consensus_queue.cc:260] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718c03482fd43918a2868abf0f8a6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } }
I20260923 23:09:32.294945 26252 raft_consensus.cc:388] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:32.294972 26252 raft_consensus.cc:482] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:32.295051 26252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:32.296393 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:32.292990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57369 (local address 127.0.0.1:35871)
0923 23:09:32.293209 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:32.293219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:32.293256 (+ 37us) server_negotiation.cc:407] Connection header received
0923 23:09:32.294363 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:32.294373 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:32.294415 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:32.294594 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:32.295617 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:32.295623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:32.295624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:32.295631 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:32.295695 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:32.295716 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:32.295814 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:32.296089 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 23:09:32.296143 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260923 23:09:32.296644 26252 raft_consensus.cc:504] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718c03482fd43918a2868abf0f8a6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } }
I20260923 23:09:32.296775 26252 leader_election.cc:302] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d718c03482fd43918a2868abf0f8a6c3; no voters:
I20260923 23:09:32.297093 26252 leader_election.cc:288] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:32.297158 26257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:32.297410 26252 sys_catalog.cc:564] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:32.297423 26257 raft_consensus.cc:686] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [term 1 LEADER]: Becoming Leader. State: Replica: d718c03482fd43918a2868abf0f8a6c3, State: Running, Role: LEADER
I20260923 23:09:32.297619 26257 consensus_queue.cc:237] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718c03482fd43918a2868abf0f8a6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } }
I20260923 23:09:32.298141 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:32.298161 26258 sys_catalog.cc:455] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d718c03482fd43918a2868abf0f8a6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718c03482fd43918a2868abf0f8a6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } } }
I20260923 23:09:32.298256 26258 sys_catalog.cc:458] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:32.298480 26264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:32.299263 26264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:32.300130 26259 sys_catalog.cc:455] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d718c03482fd43918a2868abf0f8a6c3. Latest consensus state: current_term: 1 leader_uuid: "d718c03482fd43918a2868abf0f8a6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718c03482fd43918a2868abf0f8a6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35871 } } }
I20260923 23:09:32.300295 26259 sys_catalog.cc:458] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:32.301728 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:32.300053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44425 (local address 127.0.0.1:35871)
0923 23:09:32.300099 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:32.300101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:32.300388 (+ 287us) server_negotiation.cc:407] Connection header received
0923 23:09:32.300520 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:32.300521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:32.300527 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:32.300562 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:32.301457 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:32.301459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:32.301459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:32.301464 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:32.301482 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:32.301488 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:32.301553 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:32.301641 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 23:09:32.301657 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:09:32.302022 26264 catalog_manager.cc:1347] Generated new cluster ID: 965980924bca462cb0dbc88e4d89a40d
I20260923 23:09:32.302074 26264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:32.310706 26264 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:32.310910 26264 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:32.314878 26264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d718c03482fd43918a2868abf0f8a6c3: Generated new TSK 0
I20260923 23:09:32.315258 26264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:32.324101 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 26184
2026-09-23T23:09:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:32 dist-test-slave-j4c5 krb5kdc[26283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:32 dist-test-slave-j4c5 krb5kdc[26283](info): set up 2 sockets
Sep 23 23:09:32 dist-test-slave-j4c5 krb5kdc[26283](info): commencing operation
krb5kdc: starting...
W20260923 23:09:33.412830 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:33 dist-test-slave-j4c5 krb5kdc[26283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:33.553582 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:33.638244 26296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:33.638597 26296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:33.638639 26296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:33.639951 26296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:33.640060 26296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:33.640084 26296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:33.640105 26296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:33.640133 26296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:33.641799 26296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:33.642145 26296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:33.642403 26296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:33.647809 26301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:33.647751 26302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:33.647874 26296 server_base.cc:1034] running on GCE node
W20260923 23:09:33.647784 26304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:33.648428 26296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:33.648778 26296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:33.650007 26296 hybrid_clock.cc:648] HybridClock initialized: now 1790204973649978 us; error 44 us; skew 500 ppm
Sep 23 23:09:33 dist-test-slave-j4c5 krb5kdc[26283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:33.652987 26296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:33.653537 26296 webserver.cc:460] Webserver started at http://127.0.0.1:39115/ using document root <none> and password file <none>
I20260923 23:09:33.653779 26296 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:33.653851 26296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:33.654011 26296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:33.655144 26296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "a4cba5e0bb114c5790a59ee9e8764458"
format_stamp: "Formatted at 2026-09-23 23:09:33 on dist-test-slave-j4c5"
server_key: "98841c9c26c6e3179586b229d0785bcb"
server_key_iv: "03a00f5f4071eae48b06d432d4d6f0ee"
server_key_version: "encryptionkey@0"
I20260923 23:09:33.655529 26296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "a4cba5e0bb114c5790a59ee9e8764458"
format_stamp: "Formatted at 2026-09-23 23:09:33 on dist-test-slave-j4c5"
server_key: "98841c9c26c6e3179586b229d0785bcb"
server_key_iv: "03a00f5f4071eae48b06d432d4d6f0ee"
server_key_version: "encryptionkey@0"
I20260923 23:09:33.657065 26296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:09:33.657920 26311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:33.658136 26296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:33.658250 26296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "a4cba5e0bb114c5790a59ee9e8764458"
format_stamp: "Formatted at 2026-09-23 23:09:33 on dist-test-slave-j4c5"
server_key: "98841c9c26c6e3179586b229d0785bcb"
server_key_iv: "03a00f5f4071eae48b06d432d4d6f0ee"
server_key_version: "encryptionkey@0"
I20260923 23:09:33.658329 26296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:33.673807 26296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:33.676235 26296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:33.676407 26296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:33.680735 26296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37973
I20260923 23:09:33.680801 26363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37973 every 8 connection(s)
I20260923 23:09:33.681192 26296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:33.682266 26364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:33.684767 26364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458: Bootstrap starting.
I20260923 23:09:33.685355 26364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:33.685600 26364 log.cc:824] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:33.686391 26364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458: No bootstrap required, opened a new log
I20260923 23:09:33.688038 26364 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cba5e0bb114c5790a59ee9e8764458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } }
I20260923 23:09:33.688280 26364 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:33.688311 26364 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cba5e0bb114c5790a59ee9e8764458, State: Initialized, Role: FOLLOWER
I20260923 23:09:33.688431 26364 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cba5e0bb114c5790a59ee9e8764458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } }
I20260923 23:09:33.688500 26364 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:33.688522 26364 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:33.688560 26364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:33.689123 26364 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cba5e0bb114c5790a59ee9e8764458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } }
I20260923 23:09:33.689210 26364 leader_election.cc:302] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4cba5e0bb114c5790a59ee9e8764458; no voters:
I20260923 23:09:33.689349 26364 leader_election.cc:288] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:33.689430 26367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:33.689608 26367 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [term 1 LEADER]: Becoming Leader. State: Replica: a4cba5e0bb114c5790a59ee9e8764458, State: Running, Role: LEADER
I20260923 23:09:33.689636 26364 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:33.689757 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 26296
I20260923 23:09:33.689754 26367 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cba5e0bb114c5790a59ee9e8764458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } }
I20260923 23:09:33.689888 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:33.690119 5416 external_mini_cluster.cc:1428] Setting key b2ae36b60cecc93dbfac9803fa5271e1
I20260923 23:09:33.690122 26367 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4cba5e0bb114c5790a59ee9e8764458. Latest consensus state: current_term: 1 leader_uuid: "a4cba5e0bb114c5790a59ee9e8764458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cba5e0bb114c5790a59ee9e8764458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } } }
I20260923 23:09:33.690182 26367 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:33.690217 26368 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4cba5e0bb114c5790a59ee9e8764458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cba5e0bb114c5790a59ee9e8764458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } } }
I20260923 23:09:33.690277 26368 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:33.690819 26374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:33.691395 26374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:33.693130 26374 catalog_manager.cc:1347] Generated new cluster ID: 5ef0269f224c4801b5db7697f217d99e
I20260923 23:09:33.693182 26374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:09:33 dist-test-slave-j4c5 krb5kdc[26283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 23:09:33.704030 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:33.690804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:49677 (local address 127.0.0.1:37973)
0923 23:09:33.691157 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:33.691169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:33.691229 (+ 60us) server_negotiation.cc:407] Connection header received
0923 23:09:33.691682 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:33.691692 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:33.691768 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:33.691936 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:33.692787 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:33.694195 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:33.695270 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:33.695954 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:33.696110 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:33.699694 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:33.699702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:33.699702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:33.699824 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:33.702068 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:33.702633 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:33.702637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:33.702637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:33.702763 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:33.703044 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:33.703045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:33.703045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:33.703368 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:33.703554 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:33.703673 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 23:09:33.703778 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":68,"threads_started":1}
I20260923 23:09:33.705088 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:33.713595 26374 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:33.714278 26374 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:33.714897 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:33.706746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:42259 (local address 127.0.0.1:37973)
0923 23:09:33.706787 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:33.706789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:33.706965 (+ 176us) server_negotiation.cc:407] Connection header received
0923 23:09:33.707036 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:33.707037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:33.707047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:33.707099 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:33.707766 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:33.708616 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:33.709694 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:33.710012 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:33.710076 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:33.711757 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:33.711760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:33.711760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:33.711768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:33.713480 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:33.713999 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:33.714001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:33.714001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:33.714062 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:33.714351 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:33.714352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:33.714353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:33.714485 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:33.714597 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:33.714750 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 23:09:33.714813 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:09:33.721915 26374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4cba5e0bb114c5790a59ee9e8764458: Generated new TSK 0
I20260923 23:09:33.722154 26374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:33.729815 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 26296
W20260923 23:09:33.734362 26281 connection.cc:535] client connection to 127.0.0.1:37973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37973): Connection reset by peer (error 104)
2026-09-23T23:09:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:33 dist-test-slave-j4c5 krb5kdc[26395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:33 dist-test-slave-j4c5 krb5kdc[26395](info): set up 2 sockets
Sep 23 23:09:33 dist-test-slave-j4c5 krb5kdc[26395](info): commencing operation
krb5kdc: starting...
W20260923 23:09:34.800400 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:34 dist-test-slave-j4c5 krb5kdc[26395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:34.931401 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:35.020608 26408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:35.020887 26408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:35.020917 26408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:35.022338 26408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:35.022408 26408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:35.022423 26408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:35.022437 26408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:35.022450 26408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:35.024386 26408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:35.024701 26408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:35.024979 26408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:35.030427 26416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:35.030419 26414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:35.030387 26413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:35.030583 26408 server_base.cc:1034] running on GCE node
I20260923 23:09:35.030974 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:35.031363 26408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:35.032634 26408 hybrid_clock.cc:648] HybridClock initialized: now 1790204975032635 us; error 74 us; skew 500 ppm
Sep 23 23:09:35 dist-test-slave-j4c5 krb5kdc[26395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:35.035423 26408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:35.035897 26408 webserver.cc:460] Webserver started at http://127.0.0.1:46865/ using document root <none> and password file <none>
I20260923 23:09:35.036227 26408 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:35.036279 26408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:35.036442 26408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:35.037557 26408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "9298f56b018f47e7b44f12bc35551fb3"
format_stamp: "Formatted at 2026-09-23 23:09:35 on dist-test-slave-j4c5"
server_key: "22ea18092450c18bec0e9fcb9a9ccacb"
server_key_iv: "f1437ac19e7506f6e3e1202c84c48a03"
server_key_version: "encryptionkey@0"
I20260923 23:09:35.037933 26408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "9298f56b018f47e7b44f12bc35551fb3"
format_stamp: "Formatted at 2026-09-23 23:09:35 on dist-test-slave-j4c5"
server_key: "22ea18092450c18bec0e9fcb9a9ccacb"
server_key_iv: "f1437ac19e7506f6e3e1202c84c48a03"
server_key_version: "encryptionkey@0"
I20260923 23:09:35.039331 26408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:09:35.040257 26423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:35.040637 26408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:35.040735 26408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "9298f56b018f47e7b44f12bc35551fb3"
format_stamp: "Formatted at 2026-09-23 23:09:35 on dist-test-slave-j4c5"
server_key: "22ea18092450c18bec0e9fcb9a9ccacb"
server_key_iv: "f1437ac19e7506f6e3e1202c84c48a03"
server_key_version: "encryptionkey@0"
I20260923 23:09:35.040963 26408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:35.067626 26408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:35.069967 26408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:35.070186 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:35.074270 26408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33151
I20260923 23:09:35.074304 26475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33151 every 8 connection(s)
I20260923 23:09:35.074671 26408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:35.075742 26476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:35.077301 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 26408
I20260923 23:09:35.077440 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:35.077656 5416 external_mini_cluster.cc:1428] Setting key 08c032230e7aeba1c624b5e1b0b6e0e1
I20260923 23:09:35.078704 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3: Bootstrap starting.
I20260923 23:09:35.079334 26476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:35.079665 26476 log.cc:824] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:35.080489 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3: No bootstrap required, opened a new log
I20260923 23:09:35.082809 26476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298f56b018f47e7b44f12bc35551fb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260923 23:09:35.083002 26476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:35.083065 26476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9298f56b018f47e7b44f12bc35551fb3, State: Initialized, Role: FOLLOWER
I20260923 23:09:35.083215 26476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298f56b018f47e7b44f12bc35551fb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260923 23:09:35.083302 26476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:35.083343 26476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:35.083400 26476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:35.084110 26476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298f56b018f47e7b44f12bc35551fb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260923 23:09:35.084235 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:35.078549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:38867 (local address 127.0.0.1:33151)
0923 23:09:35.078767 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:35.078777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:35.078811 (+ 34us) server_negotiation.cc:407] Connection header received
0923 23:09:35.079260 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:35.079267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:35.079315 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:35.079486 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:35.079807 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:35.080971 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:35.082092 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:35.082494 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:35.082579 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:35.083397 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:35.083402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:35.083402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:35.083636 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:35.083669 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:35.083680 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:35.083752 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:35.083961 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 23:09:35.083997 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260923 23:09:35.084249 26476 leader_election.cc:302] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9298f56b018f47e7b44f12bc35551fb3; no voters:
I20260923 23:09:35.084522 26476 leader_election.cc:288] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:35.084561 26481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:35.084692 26481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [term 1 LEADER]: Becoming Leader. State: Replica: 9298f56b018f47e7b44f12bc35551fb3, State: Running, Role: LEADER
I20260923 23:09:35.084815 26476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:35.084823 26481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298f56b018f47e7b44f12bc35551fb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260923 23:09:35.085263 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:35.085369 26483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9298f56b018f47e7b44f12bc35551fb3. Latest consensus state: current_term: 1 leader_uuid: "9298f56b018f47e7b44f12bc35551fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298f56b018f47e7b44f12bc35551fb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } } }
I20260923 23:09:35.085439 26483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [sys.catalog]: This master's current role is: LEADER
W20260923 23:09:35.086810 26497 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:35.086926 26497 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:35.086966 26482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9298f56b018f47e7b44f12bc35551fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9298f56b018f47e7b44f12bc35551fb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } } }
I20260923 23:09:35.087069 26490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:35.087153 26482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:35.087306 26490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:35.089798 26490 catalog_manager.cc:1347] Generated new cluster ID: 3238dd13056e4723853820ab9f06c26c
I20260923 23:09:35.089869 26490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:35.090852 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:35.087032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:56567 (local address 127.0.0.1:33151)
0923 23:09:35.087096 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:35.087098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:35.087411 (+ 313us) server_negotiation.cc:407] Connection header received
0923 23:09:35.087462 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:35.087463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:35.087474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:35.087545 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:35.087711 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:35.088376 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:35.089461 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:35.089661 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:35.089764 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:35.090610 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:35.090611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:35.090612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:35.090617 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:35.090631 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:35.090636 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:35.090691 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:35.090752 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 23:09:35.090761 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 23:09:35.104295 26490 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:35.104987 26490 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:35.112264 26490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9298f56b018f47e7b44f12bc35551fb3: Generated new TSK 0
I20260923 23:09:35.112617 26490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:35.131717 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 26408
W20260923 23:09:35.137228 26393 connection.cc:535] client connection to 127.0.0.1:33151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33151): Connection reset by peer (error 104)
2026-09-23T23:09:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:35 dist-test-slave-j4c5 krb5kdc[26507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:35 dist-test-slave-j4c5 krb5kdc[26507](info): set up 2 sockets
Sep 23 23:09:35 dist-test-slave-j4c5 krb5kdc[26507](info): commencing operation
krb5kdc: starting...
W20260923 23:09:36.196503 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:36 dist-test-slave-j4c5 krb5kdc[26507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:36.328411 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:36.413595 26520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:36.413815 26520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:36.413841 26520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:36.415074 26520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:36.415129 26520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:36.415144 26520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:36.415158 26520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:36.415169 26520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:36.416831 26520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:36.417106 26520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:36.417341 26520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:36.422713 26528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:36.422742 26526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:36.422742 26525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:36.423085 26520 server_base.cc:1034] running on GCE node
I20260923 23:09:36.423395 26520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:36.423698 26520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:36.424866 26520 hybrid_clock.cc:648] HybridClock initialized: now 1790204976424854 us; error 36 us; skew 500 ppm
Sep 23 23:09:36 dist-test-slave-j4c5 krb5kdc[26507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:36.427666 26520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:36.428247 26520 webserver.cc:460] Webserver started at http://127.0.0.1:46561/ using document root <none> and password file <none>
I20260923 23:09:36.428551 26520 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:36.428628 26520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:36.428781 26520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:36.429899 26520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205"
format_stamp: "Formatted at 2026-09-23 23:09:36 on dist-test-slave-j4c5"
server_key: "9415d2ee6e1544cb81d059482f4abe79"
server_key_iv: "1f859a8be0a41c2bf6af8827a408974f"
server_key_version: "encryptionkey@0"
I20260923 23:09:36.430275 26520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205"
format_stamp: "Formatted at 2026-09-23 23:09:36 on dist-test-slave-j4c5"
server_key: "9415d2ee6e1544cb81d059482f4abe79"
server_key_iv: "1f859a8be0a41c2bf6af8827a408974f"
server_key_version: "encryptionkey@0"
I20260923 23:09:36.431663 26520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:09:36.432541 26535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:36.432796 26520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:36.432904 26520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205"
format_stamp: "Formatted at 2026-09-23 23:09:36 on dist-test-slave-j4c5"
server_key: "9415d2ee6e1544cb81d059482f4abe79"
server_key_iv: "1f859a8be0a41c2bf6af8827a408974f"
server_key_version: "encryptionkey@0"
I20260923 23:09:36.432992 26520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:36.436579 26520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:36.438763 26520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:36.438932 26520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:36.443574 26520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33307
I20260923 23:09:36.443603 26587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33307 every 8 connection(s)
I20260923 23:09:36.443953 26520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:36.444995 26588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:36.447350 26588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205: Bootstrap starting.
I20260923 23:09:36.447956 26588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:36.448310 26588 log.cc:824] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:36.449121 26588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205: No bootstrap required, opened a new log
I20260923 23:09:36.450865 26588 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } }
I20260923 23:09:36.451035 26588 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:36.451081 26588 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a61f64a0794bb9a93ebfbf0ddf3205, State: Initialized, Role: FOLLOWER
I20260923 23:09:36.451296 26588 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } }
I20260923 23:09:36.451397 26588 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:36.451443 26588 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:36.451498 26588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:36.452078 26588 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } }
I20260923 23:09:36.452234 26588 leader_election.cc:302] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3a61f64a0794bb9a93ebfbf0ddf3205; no voters:
I20260923 23:09:36.452442 26588 leader_election.cc:288] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:36.452509 26591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:36.452710 26591 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [term 1 LEADER]: Becoming Leader. State: Replica: b3a61f64a0794bb9a93ebfbf0ddf3205, State: Running, Role: LEADER
I20260923 23:09:36.452725 26588 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:36.452821 26591 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } }
I20260923 23:09:36.453469 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } } }
I20260923 23:09:36.453491 26593 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3a61f64a0794bb9a93ebfbf0ddf3205. Latest consensus state: current_term: 1 leader_uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a61f64a0794bb9a93ebfbf0ddf3205" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } } }
I20260923 23:09:36.453583 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:36.453611 26593 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:36.453776 26600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:36.453784 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 26520
I20260923 23:09:36.453905 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:36.454124 5416 external_mini_cluster.cc:1428] Setting key be3ff8c4443f6ee1abfa736205609453
I20260923 23:09:36.454342 26600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:36.456481 26600 catalog_manager.cc:1347] Generated new cluster ID: d1558184a2ba474aafaf90950d8efe78
I20260923 23:09:36.456537 26600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:36.456889 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:36.454772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:36271 (local address 127.0.0.1:33307)
0923 23:09:36.455178 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:36.455188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:36.455225 (+ 37us) server_negotiation.cc:407] Connection header received
0923 23:09:36.455348 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:36.455355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:36.455388 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:36.455552 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:36.456328 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:36.456331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:36.456331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:36.456336 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:36.456376 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:36.456391 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:36.456472 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:36.456649 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 23:09:36.456694 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":79,"threads_started":1}
I20260923 23:09:36.457804 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:36.460446 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:36.459236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:41479 (local address 127.0.0.1:33307)
0923 23:09:36.459323 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:36.459326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:36.459378 (+ 52us) server_negotiation.cc:407] Connection header received
0923 23:09:36.459443 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:36.459444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:36.459448 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:36.459513 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:36.460138 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:36.460140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:36.460140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:36.460145 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:36.460179 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:36.460185 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:36.460244 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:36.460346 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 23:09:36.460361 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 23:09:36.485845 26600 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:36.486169 26600 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:36.490531 26600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3a61f64a0794bb9a93ebfbf0ddf3205: Generated new TSK 0
I20260923 23:09:36.490811 26600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:36.500200 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 26520
2026-09-23T23:09:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:36 dist-test-slave-j4c5 krb5kdc[26619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:36 dist-test-slave-j4c5 krb5kdc[26619](info): set up 2 sockets
Sep 23 23:09:36 dist-test-slave-j4c5 krb5kdc[26619](info): commencing operation
krb5kdc: starting...
W20260923 23:09:37.548435 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:37 dist-test-slave-j4c5 krb5kdc[26619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:37.676301 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:37.761936 26632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:37.762213 26632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:37.762253 26632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:37.763507 26632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:37.763587 26632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:37.763610 26632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:37.763631 26632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:37.763664 26632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:37.765283 26632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44743
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:37.765606 26632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:37.765859 26632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:37.770921 26637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:37.770923 26638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:37.770946 26640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:37.770998 26632 server_base.cc:1034] running on GCE node
I20260923 23:09:37.771473 26632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:37.771751 26632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:37.772944 26632 hybrid_clock.cc:648] HybridClock initialized: now 1790204977772915 us; error 40 us; skew 500 ppm
Sep 23 23:09:37 dist-test-slave-j4c5 krb5kdc[26619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:37.781683 26632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:37.782223 26632 webserver.cc:460] Webserver started at http://127.0.0.1:46731/ using document root <none> and password file <none>
I20260923 23:09:37.782506 26632 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:37.782585 26632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:37.782742 26632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:37.783897 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "4342294d312249219008fa35e2484230"
format_stamp: "Formatted at 2026-09-23 23:09:37 on dist-test-slave-j4c5"
server_key: "0dac2370ae68007fa8f856480b98d38b"
server_key_iv: "22a11be81a500a7cdea84119a4c4ad2b"
server_key_version: "encryptionkey@0"
I20260923 23:09:37.784358 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "4342294d312249219008fa35e2484230"
format_stamp: "Formatted at 2026-09-23 23:09:37 on dist-test-slave-j4c5"
server_key: "0dac2370ae68007fa8f856480b98d38b"
server_key_iv: "22a11be81a500a7cdea84119a4c4ad2b"
server_key_version: "encryptionkey@0"
I20260923 23:09:37.785856 26632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:37.786761 26647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:37.786968 26632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:37.787077 26632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "4342294d312249219008fa35e2484230"
format_stamp: "Formatted at 2026-09-23 23:09:37 on dist-test-slave-j4c5"
server_key: "0dac2370ae68007fa8f856480b98d38b"
server_key_iv: "22a11be81a500a7cdea84119a4c4ad2b"
server_key_version: "encryptionkey@0"
I20260923 23:09:37.787168 26632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:37.810967 26632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:37.813453 26632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:37.813655 26632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:37.817823 26632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44743
I20260923 23:09:37.817888 26699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44743 every 8 connection(s)
I20260923 23:09:37.818235 26632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:37.819234 26700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:37.822026 26700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230: Bootstrap starting.
I20260923 23:09:37.822221 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 26632
I20260923 23:09:37.822439 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:37.822664 5416 external_mini_cluster.cc:1428] Setting key 2786095a84422a5582d27c6221b2f9a1
I20260923 23:09:37.822782 26700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:37.823174 26700 log.cc:824] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:37.824138 26700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230: No bootstrap required, opened a new log
I20260923 23:09:37.826578 26700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4342294d312249219008fa35e2484230" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44743 } }
I20260923 23:09:37.826774 26700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:37.826808 26700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4342294d312249219008fa35e2484230, State: Initialized, Role: FOLLOWER
I20260923 23:09:37.826954 26700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4342294d312249219008fa35e2484230" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44743 } }
I20260923 23:09:37.827050 26700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:37.827082 26700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:37.827132 26700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:37.827919 26700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4342294d312249219008fa35e2484230" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44743 } }
I20260923 23:09:37.828035 26700 leader_election.cc:302] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4342294d312249219008fa35e2484230; no voters:
I20260923 23:09:37.828233 26700 leader_election.cc:288] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:37.828311 26705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:37.828508 26705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [term 1 LEADER]: Becoming Leader. State: Replica: 4342294d312249219008fa35e2484230, State: Running, Role: LEADER
I20260923 23:09:37.828617 26705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4342294d312249219008fa35e2484230" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44743 } }
I20260923 23:09:37.828522 26700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:37.829046 26707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4342294d312249219008fa35e2484230. Latest consensus state: current_term: 1 leader_uuid: "4342294d312249219008fa35e2484230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4342294d312249219008fa35e2484230" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44743 } } }
I20260923 23:09:37.829180 26707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:37.829667 26706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4342294d312249219008fa35e2484230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4342294d312249219008fa35e2484230" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44743 } } }
I20260923 23:09:37.829756 26706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230 [sys.catalog]: This master's current role is: LEADER
Sep 23 23:09:37 dist-test-slave-j4c5 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260923 23:09:37.830737 26720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:37.831398 26720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:37.831959 26721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:37.832297 26721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:37.834314 26721 catalog_manager.cc:1347] Generated new cluster ID: 704514827e2e46b89fcef3982de1056b
I20260923 23:09:37.834458 26721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:37.836280 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:37.823338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:60499 (local address 127.0.0.1:44743)
0923 23:09:37.823606 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:37.823614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:37.823641 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:09:37.824349 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:37.824359 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:37.824437 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:37.824600 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:37.825311 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:37.826811 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:37.827966 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:37.828622 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:37.828743 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:37.831936 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:37.831942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:37.831942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:37.831958 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:37.834237 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:37.834769 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:37.834772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:37.834773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:37.834880 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:37.835151 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:37.835152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:37.835152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:37.835453 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:37.835599 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:37.835753 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 23:09:37.835852 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":177,"threads_started":1}
I20260923 23:09:37.837268 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:37.842794 26721 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:37.843348 26721 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:37.846887 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:37.838810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:36965 (local address 127.0.0.1:44743)
0923 23:09:37.838854 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:37.838856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:37.839008 (+ 152us) server_negotiation.cc:407] Connection header received
0923 23:09:37.839066 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:37.839067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:37.839078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:37.839133 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:37.839808 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:37.840582 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:37.841622 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:37.841967 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:37.842039 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:37.843874 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:37.843876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:37.843876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:37.843884 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:37.845605 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:37.846069 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:37.846070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:37.846070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:37.846130 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:37.846357 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:37.846357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:37.846358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:37.846477 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:37.846569 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:37.846726 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 23:09:37.846791 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:09:37.851891 26721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4342294d312249219008fa35e2484230: Generated new TSK 0
I20260923 23:09:37.852133 26721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:37.858210 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 26632
W20260923 23:09:37.862453 26617 connection.cc:535] client connection to 127.0.0.1:44743 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44743): Connection reset by peer (error 104)
2026-09-23T23:09:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:37 dist-test-slave-j4c5 krb5kdc[26731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:37 dist-test-slave-j4c5 krb5kdc[26731](info): set up 2 sockets
Sep 23 23:09:37 dist-test-slave-j4c5 krb5kdc[26731](info): commencing operation
krb5kdc: starting...
W20260923 23:09:38.904865 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:38 dist-test-slave-j4c5 krb5kdc[26731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:39.044047 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:39.135367 26744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:39.135625 26744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:39.135667 26744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:39.137025 26744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:39.137100 26744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:39.137127 26744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:39.137159 26744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:39.137185 26744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:39.138788 26744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:39.139096 26744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:39.139351 26744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:39.144507 26749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:39.144548 26752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:39.144629 26750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:39.145025 26744 server_base.cc:1034] running on GCE node
I20260923 23:09:39.145282 26744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:39.145629 26744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:39.146791 26744 hybrid_clock.cc:648] HybridClock initialized: now 1790204979146768 us; error 46 us; skew 500 ppm
Sep 23 23:09:39 dist-test-slave-j4c5 krb5kdc[26731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:39.149679 26744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:39.150251 26744 webserver.cc:460] Webserver started at http://127.0.0.1:45951/ using document root <none> and password file <none>
I20260923 23:09:39.150514 26744 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:39.150600 26744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:39.150764 26744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:39.151965 26744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "807413ca02624421bd9d8a299f673c6d"
format_stamp: "Formatted at 2026-09-23 23:09:39 on dist-test-slave-j4c5"
server_key: "8373a5687850c3c77ef2f1afb586f6a7"
server_key_iv: "b9f43b3ee10547d651c0aed701798719"
server_key_version: "encryptionkey@0"
I20260923 23:09:39.152428 26744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "807413ca02624421bd9d8a299f673c6d"
format_stamp: "Formatted at 2026-09-23 23:09:39 on dist-test-slave-j4c5"
server_key: "8373a5687850c3c77ef2f1afb586f6a7"
server_key_iv: "b9f43b3ee10547d651c0aed701798719"
server_key_version: "encryptionkey@0"
I20260923 23:09:39.154052 26744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:39.155107 26759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:39.155303 26744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:39.155426 26744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "807413ca02624421bd9d8a299f673c6d"
format_stamp: "Formatted at 2026-09-23 23:09:39 on dist-test-slave-j4c5"
server_key: "8373a5687850c3c77ef2f1afb586f6a7"
server_key_iv: "b9f43b3ee10547d651c0aed701798719"
server_key_version: "encryptionkey@0"
I20260923 23:09:39.155541 26744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:39.177454 26744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:39.179884 26744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:39.180061 26744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:39.184082 26744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32985
I20260923 23:09:39.184136 26811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32985 every 8 connection(s)
I20260923 23:09:39.184526 26744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:39.185444 26812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:39.187680 26812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d: Bootstrap starting.
I20260923 23:09:39.188330 26812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:39.188570 26812 log.cc:824] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:39.189236 26812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d: No bootstrap required, opened a new log
I20260923 23:09:39.189916 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 26744
I20260923 23:09:39.190205 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:39.190495 5416 external_mini_cluster.cc:1428] Setting key a9598f42527ae9ed54d8db859facdc8d
I20260923 23:09:39.191165 26812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807413ca02624421bd9d8a299f673c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260923 23:09:39.191355 26812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:39.191382 26812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807413ca02624421bd9d8a299f673c6d, State: Initialized, Role: FOLLOWER
I20260923 23:09:39.191541 26812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807413ca02624421bd9d8a299f673c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260923 23:09:39.191604 26812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:39.191627 26812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:39.191695 26812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:39.192688 26812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807413ca02624421bd9d8a299f673c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260923 23:09:39.192808 26812 leader_election.cc:302] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 807413ca02624421bd9d8a299f673c6d; no voters:
I20260923 23:09:39.193006 26812 leader_election.cc:288] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:39.193027 26817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:39.193161 26817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [term 1 LEADER]: Becoming Leader. State: Replica: 807413ca02624421bd9d8a299f673c6d, State: Running, Role: LEADER
I20260923 23:09:39.193295 26812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:39.193295 26817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807413ca02624421bd9d8a299f673c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260923 23:09:39.193758 26819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 807413ca02624421bd9d8a299f673c6d. Latest consensus state: current_term: 1 leader_uuid: "807413ca02624421bd9d8a299f673c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807413ca02624421bd9d8a299f673c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } } }
I20260923 23:09:39.193845 26819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:39.194058 26818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "807413ca02624421bd9d8a299f673c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807413ca02624421bd9d8a299f673c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } } }
I20260923 23:09:39.194180 26818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:39.194864 26823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:39.195537 26823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:39.197081 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:39.191113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:41251 (local address 127.0.0.1:32985)
0923 23:09:39.191309 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:39.191319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:39.191345 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:09:39.191509 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:39.191517 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:39.191582 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:39.191738 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:39.192071 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:39.193188 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:39.194382 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:39.195236 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:39.195379 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:39.196231 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:39.196235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:39.196235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:39.196438 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:39.196479 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:39.196495 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:39.196770 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:39.196820 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 23:09:39.196849 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260923 23:09:39.197227 26823 catalog_manager.cc:1347] Generated new cluster ID: 30548b84c82147eabce3bba6ea04a46f
I20260923 23:09:39.197479 26823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:39.198284 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:39.202984 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:39.199689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:36217 (local address 127.0.0.1:32985)
0923 23:09:39.199752 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:39.199753 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:39.199869 (+ 116us) server_negotiation.cc:407] Connection header received
0923 23:09:39.199943 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:39.199944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:39.199956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:39.200000 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:39.200192 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:39.200677 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:39.201389 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:39.201890 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:39.201975 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:39.202706 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:39.202708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:39.202709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:39.202714 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:39.202729 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:39.202734 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:39.202792 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:39.202867 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 23:09:39.202878 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:09:39.212628 26823 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:39.213557 26823 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:39.220042 26823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 807413ca02624421bd9d8a299f673c6d: Generated new TSK 0
I20260923 23:09:39.220299 26823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:39.227757 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 26744
W20260923 23:09:39.232692 26729 connection.cc:535] client connection to 127.0.0.1:32985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32985): Connection reset by peer (error 104)
2026-09-23T23:09:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:39 dist-test-slave-j4c5 krb5kdc[26843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:39 dist-test-slave-j4c5 krb5kdc[26843](info): set up 2 sockets
Sep 23 23:09:39 dist-test-slave-j4c5 krb5kdc[26843](info): commencing operation
krb5kdc: starting...
W20260923 23:09:40.344472 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:40 dist-test-slave-j4c5 krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:09:40.480419 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:40.568184 26856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:40.568420 26856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:40.568446 26856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:40.569661 26856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:40.569702 26856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:40.569717 26856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:40.569729 26856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:40.569742 26856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:40.571223 26856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:40.571432 26856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:40.571623 26856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:40.576766 26862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:40.576850 26861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:40.576874 26864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:40.576831 26856 server_base.cc:1034] running on GCE node
I20260923 23:09:40.577260 26856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:40.577593 26856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:40.578768 26856 hybrid_clock.cc:648] HybridClock initialized: now 1790204980578757 us; error 46 us; skew 500 ppm
Sep 23 23:09:40 dist-test-slave-j4c5 krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:40.581607 26856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:09:40.582209 26856 webserver.cc:460] Webserver started at http://127.0.0.1:39129/ using document root <none> and password file <none>
I20260923 23:09:40.582480 26856 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:40.582557 26856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:40.582765 26856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:40.583961 26856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "6a79a732be914033a88e630e2c618f36"
format_stamp: "Formatted at 2026-09-23 23:09:40 on dist-test-slave-j4c5"
server_key: "76fc2b9188ca52f3d99e65f6be96c108"
server_key_iv: "7d266c011ddee18687943f520e63c96f"
server_key_version: "encryptionkey@0"
I20260923 23:09:40.584398 26856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "6a79a732be914033a88e630e2c618f36"
format_stamp: "Formatted at 2026-09-23 23:09:40 on dist-test-slave-j4c5"
server_key: "76fc2b9188ca52f3d99e65f6be96c108"
server_key_iv: "7d266c011ddee18687943f520e63c96f"
server_key_version: "encryptionkey@0"
I20260923 23:09:40.585932 26856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:40.586751 26871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:40.586995 26856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:09:40.587093 26856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "6a79a732be914033a88e630e2c618f36"
format_stamp: "Formatted at 2026-09-23 23:09:40 on dist-test-slave-j4c5"
server_key: "76fc2b9188ca52f3d99e65f6be96c108"
server_key_iv: "7d266c011ddee18687943f520e63c96f"
server_key_version: "encryptionkey@0"
I20260923 23:09:40.587172 26856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:40.590767 26856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:40.593214 26856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:40.593398 26856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:40.597283 26856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37519
I20260923 23:09:40.597355 26923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37519 every 8 connection(s)
I20260923 23:09:40.597716 26856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:40.598875 26924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:40.601400 26924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36: Bootstrap starting.
I20260923 23:09:40.602164 26924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:40.602483 26924 log.cc:824] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:40.603330 26924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36: No bootstrap required, opened a new log
I20260923 23:09:40.605110 26924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a79a732be914033a88e630e2c618f36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } }
I20260923 23:09:40.605263 26924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:40.605285 26924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a79a732be914033a88e630e2c618f36, State: Initialized, Role: FOLLOWER
I20260923 23:09:40.605448 26924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a79a732be914033a88e630e2c618f36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } }
I20260923 23:09:40.605520 26924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:40.605542 26924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:40.605590 26924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:40.606231 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 26856
I20260923 23:09:40.606287 26924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a79a732be914033a88e630e2c618f36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } }
I20260923 23:09:40.606366 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:40.606395 26924 leader_election.cc:302] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a79a732be914033a88e630e2c618f36; no voters:
I20260923 23:09:40.606580 5416 external_mini_cluster.cc:1428] Setting key 5cd601bba2e078d9f3b44fdc94bceb22
I20260923 23:09:40.606585 26924 leader_election.cc:288] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:40.606658 26927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:40.606863 26924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:40.606853 26927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [term 1 LEADER]: Becoming Leader. State: Replica: 6a79a732be914033a88e630e2c618f36, State: Running, Role: LEADER
I20260923 23:09:40.607153 26927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a79a732be914033a88e630e2c618f36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } }
I20260923 23:09:40.607841 26928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a79a732be914033a88e630e2c618f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a79a732be914033a88e630e2c618f36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } } }
I20260923 23:09:40.607946 26928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:40.608531 26929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a79a732be914033a88e630e2c618f36. Latest consensus state: current_term: 1 leader_uuid: "6a79a732be914033a88e630e2c618f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a79a732be914033a88e630e2c618f36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } } }
I20260923 23:09:40.608696 26929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:40.608747 26944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:40.609258 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:40.607091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:43205 (local address 127.0.0.1:37519)
0923 23:09:40.607354 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:40.607363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:40.607456 (+ 93us) server_negotiation.cc:407] Connection header received
0923 23:09:40.607583 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:40.607591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:40.607623 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:40.608120 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:40.608695 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:40.608698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:40.608698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:40.608703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:40.608746 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:40.608763 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:40.608992 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:40.609026 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 23:09:40.609074 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20260923 23:09:40.609490 26944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:40.610318 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:40.611231 26944 catalog_manager.cc:1347] Generated new cluster ID: 9e5b87244a484f70b3e99acaebc2e607
I20260923 23:09:40.611292 26944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:40.613044 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:40.611857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:35107 (local address 127.0.0.1:37519)
0923 23:09:40.611924 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:40.611926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:40.612050 (+ 124us) server_negotiation.cc:407] Connection header received
0923 23:09:40.612082 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:40.612083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:40.612089 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:40.612192 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:40.612820 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:40.612821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:40.612822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:40.612826 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:40.612842 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:40.612847 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:40.612910 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:40.612963 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 23:09:40.612977 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 23:09:40.616096 26944 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:40.616314 26944 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:40.626240 26944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a79a732be914033a88e630e2c618f36: Generated new TSK 0
I20260923 23:09:40.626477 26944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:40.628750 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 26856
2026-09-23T23:09:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:40 dist-test-slave-j4c5 krb5kdc[26956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:40 dist-test-slave-j4c5 krb5kdc[26956](info): set up 2 sockets
Sep 23 23:09:40 dist-test-slave-j4c5 krb5kdc[26956](info): commencing operation
krb5kdc: starting...
W20260923 23:09:41.692574 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:41 dist-test-slave-j4c5 krb5kdc[26956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:41.839465 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35357
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:41.925685 26969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:41.925967 26969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:41.926008 26969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:41.927346 26969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:41.927438 26969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:41.927464 26969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:41.927486 26969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:41.927523 26969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:41.929243 26969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:35357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:41.929576 26969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:41.929831 26969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:41.936131 26974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:41.936189 26977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:41.936419 26969 server_base.cc:1034] running on GCE node
W20260923 23:09:41.936131 26975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:41.936739 26969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:41.937075 26969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:41.938248 26969 hybrid_clock.cc:648] HybridClock initialized: now 1790204981938235 us; error 33 us; skew 500 ppm
Sep 23 23:09:41 dist-test-slave-j4c5 krb5kdc[26956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204981, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:41.941298 26969 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:41.941857 26969 webserver.cc:460] Webserver started at http://127.5.74.62:35157/ using document root <none> and password file <none>
I20260923 23:09:41.942137 26969 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:41.942210 26969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:41.942440 26969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:41.943575 26969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "12084753db144820afa60272e6e69bf8"
format_stamp: "Formatted at 2026-09-23 23:09:41 on dist-test-slave-j4c5"
server_key: "99d8afb73380294983747ad1c446771f"
server_key_iv: "5d2033170d36199de2e9115c49446264"
server_key_version: "encryptionkey@0"
I20260923 23:09:41.943975 26969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "12084753db144820afa60272e6e69bf8"
format_stamp: "Formatted at 2026-09-23 23:09:41 on dist-test-slave-j4c5"
server_key: "99d8afb73380294983747ad1c446771f"
server_key_iv: "5d2033170d36199de2e9115c49446264"
server_key_version: "encryptionkey@0"
I20260923 23:09:41.945557 26969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:41.946374 26984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:41.946650 26969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:09:41.946763 26969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "12084753db144820afa60272e6e69bf8"
format_stamp: "Formatted at 2026-09-23 23:09:41 on dist-test-slave-j4c5"
server_key: "99d8afb73380294983747ad1c446771f"
server_key_iv: "5d2033170d36199de2e9115c49446264"
server_key_version: "encryptionkey@0"
I20260923 23:09:41.946866 26969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:41.959944 26969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:41.962778 26969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:41.963011 26969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:41.967464 26969 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:35357
I20260923 23:09:41.967505 27036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:35357 every 8 connection(s)
I20260923 23:09:41.967888 26969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:41.968858 27037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:41.971238 27037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8: Bootstrap starting.
I20260923 23:09:41.971868 27037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:41.972237 27037 log.cc:824] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:41.973083 27037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8: No bootstrap required, opened a new log
I20260923 23:09:41.974825 27037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12084753db144820afa60272e6e69bf8" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35357 } }
I20260923 23:09:41.974982 27037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:41.975005 27037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12084753db144820afa60272e6e69bf8, State: Initialized, Role: FOLLOWER
I20260923 23:09:41.975173 27037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12084753db144820afa60272e6e69bf8" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35357 } }
I20260923 23:09:41.975239 27037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:41.975260 27037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:41.975287 27037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:41.976130 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 26969
I20260923 23:09:41.976318 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:41.976344 27037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12084753db144820afa60272e6e69bf8" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35357 } }
I20260923 23:09:41.976491 27037 leader_election.cc:302] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12084753db144820afa60272e6e69bf8; no voters:
I20260923 23:09:41.976572 5416 external_mini_cluster.cc:1428] Setting key b3f2859d19aa0363a95e50fbee6c5d35
I20260923 23:09:41.976707 27037 leader_election.cc:288] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:41.976735 27040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:41.976924 27040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [term 1 LEADER]: Becoming Leader. State: Replica: 12084753db144820afa60272e6e69bf8, State: Running, Role: LEADER
I20260923 23:09:41.977036 27037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:41.977064 27040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12084753db144820afa60272e6e69bf8" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35357 } }
I20260923 23:09:41.977487 27041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12084753db144820afa60272e6e69bf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12084753db144820afa60272e6e69bf8" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35357 } } }
I20260923 23:09:41.977578 27041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:41.977716 27041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12084753db144820afa60272e6e69bf8. Latest consensus state: current_term: 1 leader_uuid: "12084753db144820afa60272e6e69bf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12084753db144820afa60272e6e69bf8" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35357 } } }
I20260923 23:09:41.977766 27041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:41.978308 27050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:41.978897 27050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:41.980585 27050 catalog_manager.cc:1347] Generated new cluster ID: 1c978020b405457382632001cd950677
I20260923 23:09:41.980660 27050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:09:41 dist-test-slave-j4c5 krb5kdc[26956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:41.989857 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:41.977206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41331 (local address 127.5.74.62:35357)
0923 23:09:41.977383 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:41.977393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:41.977449 (+ 56us) server_negotiation.cc:407] Connection header received
0923 23:09:41.977615 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:41.977623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:41.977684 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:41.978479 (+ 795us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:41.978870 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:41.980098 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:41.981266 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:41.981999 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:41.982116 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:41.985126 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:41.985131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:41.985132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:41.985146 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:41.987916 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:41.988468 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:41.988473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:41.988473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:41.988613 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:41.988860 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:41.988861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:41.988861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:41.989209 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:41.989405 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:41.989490 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 23:09:41.989609 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260923 23:09:41.993943 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:41.998051 27050 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:41.998661 27050 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:42.002506 27050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12084753db144820afa60272e6e69bf8: Generated new TSK 0
I20260923 23:09:42.002737 27050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:42.003484 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:41.995571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60365 (local address 127.5.74.62:35357)
0923 23:09:41.995623 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:41.995625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:41.995764 (+ 139us) server_negotiation.cc:407] Connection header received
0923 23:09:41.995815 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:41.995816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:41.995827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:41.995893 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:41.996605 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:41.997387 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:41.998374 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:41.998694 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:41.998762 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:42.000361 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:42.000363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:42.000363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:42.000370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:42.001913 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:42.002511 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:42.002512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:42.002513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:42.002581 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:42.002876 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:42.002877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:42.002877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:42.003005 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:42.003082 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:42.003322 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 23:09:42.003379 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:09:42.005686 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 26969
W20260923 23:09:42.010221 26954 connection.cc:535] client connection to 127.5.74.62:35357 recv error: Network error: failed to read from TLS socket (remote: 127.5.74.62:35357): Connection reset by peer (error 104)
2026-09-23T23:09:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:42 dist-test-slave-j4c5 krb5kdc[27068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:42 dist-test-slave-j4c5 krb5kdc[27068](info): set up 2 sockets
Sep 23 23:09:42 dist-test-slave-j4c5 krb5kdc[27068](info): commencing operation
krb5kdc: starting...
W20260923 23:09:43.080549 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:43 dist-test-slave-j4c5 krb5kdc[27068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:43.213644 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:40131
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:40131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:43.299427 27081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:43.299696 27081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:43.299734 27081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:43.301015 27081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:43.301090 27081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:43.301115 27081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:43.301136 27081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:43.301167 27081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:43.302685 27081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:40131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:40131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:43.303104 27081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:43.303391 27081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:43.308881 27086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:43.308881 27089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:43.308929 27087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:43.309258 27081 server_base.cc:1034] running on GCE node
I20260923 23:09:43.309520 27081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:43.309788 27081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:43.310961 27081 hybrid_clock.cc:648] HybridClock initialized: now 1790204983310936 us; error 41 us; skew 500 ppm
Sep 23 23:09:43 dist-test-slave-j4c5 krb5kdc[27068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204983, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:43.320401 27081 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:43.320899 27081 webserver.cc:460] Webserver started at http://127.5.74.62:41317/ using document root <none> and password file <none>
I20260923 23:09:43.321131 27081 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:43.321178 27081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:43.321310 27081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:43.322402 27081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "ddeb113ac5bc46cba31175c86a0d702e"
format_stamp: "Formatted at 2026-09-23 23:09:43 on dist-test-slave-j4c5"
server_key: "a0bfb891f0656426bae59a3b5ebb5942"
server_key_iv: "08791419a9649d77364cb344de6ce122"
server_key_version: "encryptionkey@0"
I20260923 23:09:43.322764 27081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "ddeb113ac5bc46cba31175c86a0d702e"
format_stamp: "Formatted at 2026-09-23 23:09:43 on dist-test-slave-j4c5"
server_key: "a0bfb891f0656426bae59a3b5ebb5942"
server_key_iv: "08791419a9649d77364cb344de6ce122"
server_key_version: "encryptionkey@0"
I20260923 23:09:43.324257 27081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 23:09:43.325235 27096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:43.325487 27081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:43.325609 27081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "ddeb113ac5bc46cba31175c86a0d702e"
format_stamp: "Formatted at 2026-09-23 23:09:43 on dist-test-slave-j4c5"
server_key: "a0bfb891f0656426bae59a3b5ebb5942"
server_key_iv: "08791419a9649d77364cb344de6ce122"
server_key_version: "encryptionkey@0"
I20260923 23:09:43.325695 27081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:43.341091 27081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:43.343636 27081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:43.343823 27081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:43.348512 27081 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:40131
I20260923 23:09:43.348665 27148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:40131 every 8 connection(s)
I20260923 23:09:43.349023 27081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:43.349524 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 27081
I20260923 23:09:43.349663 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:43.349860 5416 external_mini_cluster.cc:1428] Setting key 8a9592bbda4f4e0c90cfb01174917368
I20260923 23:09:43.350260 27149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:43.354318 27149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e: Bootstrap starting.
I20260923 23:09:43.355142 27149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:43.355521 27149 log.cc:824] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:43.356401 27149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e: No bootstrap required, opened a new log
I20260923 23:09:43.356606 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:43.350481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56165 (local address 127.5.74.62:40131)
0923 23:09:43.350705 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:43.350713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:43.350743 (+ 30us) server_negotiation.cc:407] Connection header received
0923 23:09:43.351219 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:43.351227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:43.351278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:43.351420 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:43.351955 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:43.353096 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:43.354275 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:43.354758 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:43.354848 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:43.355741 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:43.355745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:43.355745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:43.355936 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:43.355975 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:43.355986 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:43.356076 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:43.356290 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 23:09:43.356324 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260923 23:09:43.357692 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:43.358963 27149 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddeb113ac5bc46cba31175c86a0d702e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40131 } }
I20260923 23:09:43.359150 27149 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:43.359190 27149 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddeb113ac5bc46cba31175c86a0d702e, State: Initialized, Role: FOLLOWER
I20260923 23:09:43.359352 27149 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddeb113ac5bc46cba31175c86a0d702e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40131 } }
I20260923 23:09:43.359469 27149 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:43.359513 27149 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:43.359577 27149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:43.360270 27149 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddeb113ac5bc46cba31175c86a0d702e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40131 } }
I20260923 23:09:43.360399 27149 leader_election.cc:302] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddeb113ac5bc46cba31175c86a0d702e; no voters:
I20260923 23:09:43.360620 27149 leader_election.cc:288] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:43.360649 27159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:43.360772 27159 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [term 1 LEADER]: Becoming Leader. State: Replica: ddeb113ac5bc46cba31175c86a0d702e, State: Running, Role: LEADER
I20260923 23:09:43.360934 27149 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:43.360921 27159 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddeb113ac5bc46cba31175c86a0d702e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40131 } }
I20260923 23:09:43.361418 27160 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddeb113ac5bc46cba31175c86a0d702e. Latest consensus state: current_term: 1 leader_uuid: "ddeb113ac5bc46cba31175c86a0d702e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddeb113ac5bc46cba31175c86a0d702e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40131 } } }
I20260923 23:09:43.361500 27160 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:43.361672 27161 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddeb113ac5bc46cba31175c86a0d702e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddeb113ac5bc46cba31175c86a0d702e" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 40131 } } }
I20260923 23:09:43.361718 27161 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:43.361719 27165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:43.362493 27165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:43.363688 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:43.359271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50061 (local address 127.5.74.62:40131)
0923 23:09:43.359340 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:43.359343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:43.359483 (+ 140us) server_negotiation.cc:407] Connection header received
0923 23:09:43.359533 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:43.359534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:43.359546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:43.359605 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:43.359816 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:43.360807 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:43.361951 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:43.362236 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:43.362687 (+ 451us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:43.363267 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:43.363269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:43.363269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:43.363274 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:43.363291 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:43.363295 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:43.363561 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:43.363590 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 23:09:43.363601 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:09:43.364488 27176 mvcc.cc:205] Tried to move back new op lower bound from 7332679611858833408 to 7332679611847139328. Current Snapshot: MvccSnapshot[applied={T|T < 7332679611858833408}]
I20260923 23:09:43.364864 27165 catalog_manager.cc:1347] Generated new cluster ID: fe077562df934beeaec43b11b2623f91
I20260923 23:09:43.364917 27165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:43.382990 27165 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:43.383854 27165 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:43.387322 27165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddeb113ac5bc46cba31175c86a0d702e: Generated new TSK 0
I20260923 23:09:43.387599 27165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:43.399251 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 27081
W20260923 23:09:43.403765 27066 connection.cc:535] client connection to 127.5.74.62:40131 recv error: Network error: failed to read from TLS socket (remote: 127.5.74.62:40131): Connection reset by peer (error 104)
2026-09-23T23:09:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:43 dist-test-slave-j4c5 krb5kdc[27180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:43 dist-test-slave-j4c5 krb5kdc[27180](info): set up 2 sockets
Sep 23 23:09:43 dist-test-slave-j4c5 krb5kdc[27180](info): commencing operation
krb5kdc: starting...
W20260923 23:09:44.485167 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:44 dist-test-slave-j4c5 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:44.628523 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35667
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:35667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:44.717792 27193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:44.718045 27193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:44.718086 27193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:44.719339 27193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:44.719419 27193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:44.719446 27193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:44.719477 27193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:44.719502 27193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:44.721144 27193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:35667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:35667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:44.721437 27193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:44.721673 27193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:44.727272 27201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:44.727404 27199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:44.727272 27198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:44.727587 27193 server_base.cc:1034] running on GCE node
I20260923 23:09:44.727948 27193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:44.728350 27193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:44.729563 27193 hybrid_clock.cc:648] HybridClock initialized: now 1790204984729526 us; error 32 us; skew 500 ppm
Sep 23 23:09:44 dist-test-slave-j4c5 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204984, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:44.732595 27193 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:44.733137 27193 webserver.cc:460] Webserver started at http://127.5.74.62:43823/ using document root <none> and password file <none>
I20260923 23:09:44.733422 27193 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:44.733530 27193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:44.733723 27193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:44.734867 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "0d10d18674134f738aa4ef8623cc55a1"
format_stamp: "Formatted at 2026-09-23 23:09:44 on dist-test-slave-j4c5"
server_key: "30a981e300a1bacdde14f4d279d1febe"
server_key_iv: "4640d8a5d565b68ff11cf8e9000d3de0"
server_key_version: "encryptionkey@0"
I20260923 23:09:44.735277 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "0d10d18674134f738aa4ef8623cc55a1"
format_stamp: "Formatted at 2026-09-23 23:09:44 on dist-test-slave-j4c5"
server_key: "30a981e300a1bacdde14f4d279d1febe"
server_key_iv: "4640d8a5d565b68ff11cf8e9000d3de0"
server_key_version: "encryptionkey@0"
I20260923 23:09:44.736928 27193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:44.737901 27208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:44.738142 27193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:44.738253 27193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "0d10d18674134f738aa4ef8623cc55a1"
format_stamp: "Formatted at 2026-09-23 23:09:44 on dist-test-slave-j4c5"
server_key: "30a981e300a1bacdde14f4d279d1febe"
server_key_iv: "4640d8a5d565b68ff11cf8e9000d3de0"
server_key_version: "encryptionkey@0"
I20260923 23:09:44.738345 27193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:44.742467 27193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:44.745105 27193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:44.745294 27193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:44.749738 27193 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:35667
I20260923 23:09:44.749773 27260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:35667 every 8 connection(s)
I20260923 23:09:44.750135 27193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:44.751092 27261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:44.753584 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1: Bootstrap starting.
I20260923 23:09:44.754199 27261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:44.754505 27261 log.cc:824] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:44.755373 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1: No bootstrap required, opened a new log
I20260923 23:09:44.755650 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 27193
I20260923 23:09:44.755795 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:44.756013 5416 external_mini_cluster.cc:1428] Setting key 1a83abc92a8b90e7f43edef853fbd494
I20260923 23:09:44.757730 27261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10d18674134f738aa4ef8623cc55a1" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35667 } }
I20260923 23:09:44.757912 27261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:44.757982 27261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d10d18674134f738aa4ef8623cc55a1, State: Initialized, Role: FOLLOWER
I20260923 23:09:44.758139 27261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10d18674134f738aa4ef8623cc55a1" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35667 } }
I20260923 23:09:44.758553 27261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:44.758611 27261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:44.758678 27261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:44.759159 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:44.756623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54231 (local address 127.5.74.62:35667)
0923 23:09:44.756813 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:44.756823 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:44.756861 (+ 38us) server_negotiation.cc:407] Connection header received
0923 23:09:44.757735 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:44.757743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:44.757773 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:44.757918 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:44.758679 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:44.758683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:44.758683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:44.758688 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:44.758755 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:44.758772 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:44.758860 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:44.758931 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 23:09:44.758975 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260923 23:09:44.759501 27261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10d18674134f738aa4ef8623cc55a1" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35667 } }
I20260923 23:09:44.759629 27261 leader_election.cc:302] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d10d18674134f738aa4ef8623cc55a1; no voters:
I20260923 23:09:44.759812 27261 leader_election.cc:288] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:44.759881 27266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:44.760025 27266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [term 1 LEADER]: Becoming Leader. State: Replica: 0d10d18674134f738aa4ef8623cc55a1, State: Running, Role: LEADER
I20260923 23:09:44.760103 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:44.760129 27261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:44.760200 27266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10d18674134f738aa4ef8623cc55a1" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35667 } }
I20260923 23:09:44.760694 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d10d18674134f738aa4ef8623cc55a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10d18674134f738aa4ef8623cc55a1" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35667 } } }
I20260923 23:09:44.760792 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:44.760982 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d10d18674134f738aa4ef8623cc55a1. Latest consensus state: current_term: 1 leader_uuid: "0d10d18674134f738aa4ef8623cc55a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10d18674134f738aa4ef8623cc55a1" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 35667 } } }
I20260923 23:09:44.761047 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:44.761953 27275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:44.762509 27275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:44.763266 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:44.761849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40457 (local address 127.5.74.62:35667)
0923 23:09:44.761906 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:44.761908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:44.762180 (+ 272us) server_negotiation.cc:407] Connection header received
0923 23:09:44.762237 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:44.762238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:44.762246 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:44.762327 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:44.763013 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:44.763015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:44.763015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:44.763020 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:44.763038 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:44.763043 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:44.763118 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:44.763180 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 23:09:44.763196 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 23:09:44.764281 27275 catalog_manager.cc:1347] Generated new cluster ID: 7907f5d39407400d860dc86a68f3a9a9
I20260923 23:09:44.764345 27275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:44.770407 27275 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:44.770646 27275 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:44.776976 27275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d10d18674134f738aa4ef8623cc55a1: Generated new TSK 0
I20260923 23:09:44.777241 27275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:44.787341 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 27193
2026-09-23T23:09:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:44 dist-test-slave-j4c5 krb5kdc[27292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:44 dist-test-slave-j4c5 krb5kdc[27292](info): set up 2 sockets
Sep 23 23:09:44 dist-test-slave-j4c5 krb5kdc[27292](info): commencing operation
krb5kdc: starting...
W20260923 23:09:45.848618 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:45 dist-test-slave-j4c5 krb5kdc[27292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:45.981690 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:38109
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:38109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:46.070536 27305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:46.070812 27305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:46.070840 27305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:46.072206 27305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:46.072252 27305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:46.072265 27305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:46.072278 27305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:46.072290 27305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:46.073798 27305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:38109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:38109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:46.074005 27305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:46.074183 27305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:46.079730 27310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:46.080113 27311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:46.080088 27313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:46.080097 27305 server_base.cc:1034] running on GCE node
I20260923 23:09:46.080546 27305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:46.080936 27305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:46.082139 27305 hybrid_clock.cc:648] HybridClock initialized: now 1790204986082130 us; error 51 us; skew 500 ppm
Sep 23 23:09:46 dist-test-slave-j4c5 krb5kdc[27292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204986, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:46.085045 27305 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:46.085558 27305 webserver.cc:460] Webserver started at http://127.5.74.62:35993/ using document root <none> and password file <none>
I20260923 23:09:46.085796 27305 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:46.085870 27305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:46.086030 27305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:46.087210 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0"
format_stamp: "Formatted at 2026-09-23 23:09:46 on dist-test-slave-j4c5"
server_key: "85d55c3513ce8629323dc6273532ecf7"
server_key_iv: "bb9055b45395c32e72d8f0c77fe43c96"
server_key_version: "encryptionkey@0"
I20260923 23:09:46.087590 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0"
format_stamp: "Formatted at 2026-09-23 23:09:46 on dist-test-slave-j4c5"
server_key: "85d55c3513ce8629323dc6273532ecf7"
server_key_iv: "bb9055b45395c32e72d8f0c77fe43c96"
server_key_version: "encryptionkey@0"
I20260923 23:09:46.089107 27305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:09:46.090098 27320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:46.090415 27305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:46.090547 27305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0"
format_stamp: "Formatted at 2026-09-23 23:09:46 on dist-test-slave-j4c5"
server_key: "85d55c3513ce8629323dc6273532ecf7"
server_key_iv: "bb9055b45395c32e72d8f0c77fe43c96"
server_key_version: "encryptionkey@0"
I20260923 23:09:46.090639 27305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:46.115175 27305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:46.117944 27305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:46.118070 27305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:46.122083 27305 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:38109
I20260923 23:09:46.122159 27372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:38109 every 8 connection(s)
I20260923 23:09:46.122556 27305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:46.123515 27373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:46.125905 27373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0: Bootstrap starting.
I20260923 23:09:46.126525 27373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:46.126814 27373 log.cc:824] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:46.127614 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 27305
I20260923 23:09:46.127650 27373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0: No bootstrap required, opened a new log
I20260923 23:09:46.127781 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:46.127985 5416 external_mini_cluster.cc:1428] Setting key afff761f39e4ac031817ec0d1f18c6dd
I20260923 23:09:46.129766 27373 raft_consensus.cc:348] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38109 } }
I20260923 23:09:46.129945 27373 raft_consensus.cc:374] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:46.130057 27373 raft_consensus.cc:729] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78afeaf2dc84cba8f7bf97bcb4bfab0, State: Initialized, Role: FOLLOWER
I20260923 23:09:46.130237 27373 consensus_queue.cc:260] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38109 } }
I20260923 23:09:46.130339 27373 raft_consensus.cc:388] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:46.130373 27373 raft_consensus.cc:482] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:46.130424 27373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:46.131443 27373 raft_consensus.cc:504] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38109 } }
I20260923 23:09:46.131538 27373 leader_election.cc:302] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c78afeaf2dc84cba8f7bf97bcb4bfab0; no voters:
I20260923 23:09:46.131718 27373 leader_election.cc:288] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:46.131742 27378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:46.131843 27378 raft_consensus.cc:686] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [term 1 LEADER]: Becoming Leader. State: Replica: c78afeaf2dc84cba8f7bf97bcb4bfab0, State: Running, Role: LEADER
I20260923 23:09:46.131992 27378 consensus_queue.cc:237] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38109 } }
I20260923 23:09:46.132004 27373 sys_catalog.cc:564] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:46.132694 27380 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c78afeaf2dc84cba8f7bf97bcb4bfab0. Latest consensus state: current_term: 1 leader_uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38109 } } }
I20260923 23:09:46.132862 27380 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:46.133147 27388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:46.133244 27379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78afeaf2dc84cba8f7bf97bcb4bfab0" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 38109 } } }
I20260923 23:09:46.133634 27379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:46.133826 27388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 23:09:46 dist-test-slave-j4c5 krb5kdc[27292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:46.136744 27388 catalog_manager.cc:1347] Generated new cluster ID: 598ba11aae99465ea124e4f2a4673034
I20260923 23:09:46.136802 27388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:46.140766 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:46.128578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38309 (local address 127.5.74.62:38109)
0923 23:09:46.128772 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:46.128780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:46.128807 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:09:46.129755 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:46.129763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:46.129820 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:46.129996 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:46.130835 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:46.131958 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:46.133070 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:46.133999 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:46.134122 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:46.137196 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:46.137200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:46.137201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:46.137213 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:46.138936 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:46.139342 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:46.139349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:46.139350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:46.139455 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:46.139684 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:46.139685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:46.139685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:46.139945 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:46.140114 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:46.140346 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 23:09:46.140603 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260923 23:09:46.141782 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:46.150122 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:46.143546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32963 (local address 127.5.74.62:38109)
0923 23:09:46.143588 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:46.143590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:46.143706 (+ 116us) server_negotiation.cc:407] Connection header received
0923 23:09:46.143814 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:46.143815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:46.143843 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:46.143927 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:46.144539 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:46.145114 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:46.145810 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:46.146055 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:46.146135 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:46.147387 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:46.147388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:46.147389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:46.147396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:46.148945 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:46.149321 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:46.149322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:46.149323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:46.149384 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:46.149595 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:46.149596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:46.149596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:46.149715 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:46.149838 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:46.149928 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 23:09:46.150009 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":21}
I20260923 23:09:46.175731 27388 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:46.176461 27388 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:46.181182 27388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c78afeaf2dc84cba8f7bf97bcb4bfab0: Generated new TSK 0
I20260923 23:09:46.181433 27388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:46.193466 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 27305
2026-09-23T23:09:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:46 dist-test-slave-j4c5 krb5kdc[27404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:46 dist-test-slave-j4c5 krb5kdc[27404](info): set up 2 sockets
Sep 23 23:09:46 dist-test-slave-j4c5 krb5kdc[27404](info): commencing operation
krb5kdc: starting...
W20260923 23:09:47.273085 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:47 dist-test-slave-j4c5 krb5kdc[27404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:47.412550 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:42529
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:42529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:47.498124 27417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:47.498373 27417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:47.498399 27417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:47.499677 27417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:47.499732 27417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:47.499747 27417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:47.499760 27417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:47.499773 27417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:47.501382 27417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:42529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:42529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:47.501672 27417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:47.501926 27417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:47.507057 27425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:47.507129 27423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:47.507133 27422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:47.507310 27417 server_base.cc:1034] running on GCE node
I20260923 23:09:47.507577 27417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:47.507836 27417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:47.509033 27417 hybrid_clock.cc:648] HybridClock initialized: now 1790204987508989 us; error 62 us; skew 500 ppm
Sep 23 23:09:47 dist-test-slave-j4c5 krb5kdc[27404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204987, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:47.511842 27417 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:47.512472 27417 webserver.cc:460] Webserver started at http://127.5.74.62:46187/ using document root <none> and password file <none>
I20260923 23:09:47.512722 27417 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:47.512789 27417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:47.512940 27417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:47.514024 27417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "3704aa69f0f749d583186b6e0c2de525"
format_stamp: "Formatted at 2026-09-23 23:09:47 on dist-test-slave-j4c5"
server_key: "3102f032c43fa05737c0705fbf680458"
server_key_iv: "66f31307181ecc42d7a9d500120cb6fa"
server_key_version: "encryptionkey@0"
I20260923 23:09:47.514382 27417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "3704aa69f0f749d583186b6e0c2de525"
format_stamp: "Formatted at 2026-09-23 23:09:47 on dist-test-slave-j4c5"
server_key: "3102f032c43fa05737c0705fbf680458"
server_key_iv: "66f31307181ecc42d7a9d500120cb6fa"
server_key_version: "encryptionkey@0"
I20260923 23:09:47.515890 27417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:47.516772 27432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:47.516989 27417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:09:47.517087 27417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "3704aa69f0f749d583186b6e0c2de525"
format_stamp: "Formatted at 2026-09-23 23:09:47 on dist-test-slave-j4c5"
server_key: "3102f032c43fa05737c0705fbf680458"
server_key_iv: "66f31307181ecc42d7a9d500120cb6fa"
server_key_version: "encryptionkey@0"
I20260923 23:09:47.517174 27417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:47.534790 27417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:47.537271 27417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:47.537458 27417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:47.542076 27417 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:42529
I20260923 23:09:47.542124 27484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:42529 every 8 connection(s)
I20260923 23:09:47.542487 27417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:47.543399 27485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:47.545644 27485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525: Bootstrap starting.
I20260923 23:09:47.546197 27485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:47.546432 27485 log.cc:824] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:47.547128 27485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525: No bootstrap required, opened a new log
I20260923 23:09:47.548812 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 27417
I20260923 23:09:47.548921 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:47.548902 27485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3704aa69f0f749d583186b6e0c2de525" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 42529 } }
I20260923 23:09:47.549114 27485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:47.549160 27485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3704aa69f0f749d583186b6e0c2de525, State: Initialized, Role: FOLLOWER
I20260923 23:09:47.549168 5416 external_mini_cluster.cc:1428] Setting key 1b28da18ee158a7d1dea5a7595422e72
I20260923 23:09:47.549306 27485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3704aa69f0f749d583186b6e0c2de525" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 42529 } }
I20260923 23:09:47.549396 27485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:47.549439 27485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:47.549489 27485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:47.550642 27485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3704aa69f0f749d583186b6e0c2de525" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 42529 } }
I20260923 23:09:47.550760 27485 leader_election.cc:302] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3704aa69f0f749d583186b6e0c2de525; no voters:
I20260923 23:09:47.550921 27485 leader_election.cc:288] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:47.550952 27490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:47.551143 27490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [term 1 LEADER]: Becoming Leader. State: Replica: 3704aa69f0f749d583186b6e0c2de525, State: Running, Role: LEADER
I20260923 23:09:47.551204 27485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:47.551307 27490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3704aa69f0f749d583186b6e0c2de525" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 42529 } }
I20260923 23:09:47.552527 27492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3704aa69f0f749d583186b6e0c2de525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3704aa69f0f749d583186b6e0c2de525" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 42529 } } }
I20260923 23:09:47.552626 27492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:47.552907 27503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:47.553056 27491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3704aa69f0f749d583186b6e0c2de525. Latest consensus state: current_term: 1 leader_uuid: "3704aa69f0f749d583186b6e0c2de525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3704aa69f0f749d583186b6e0c2de525" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 42529 } } }
I20260923 23:09:47.553146 27491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:47.553426 27503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:47.555111 27503 catalog_manager.cc:1347] Generated new cluster ID: 3cf6d4177c46454e8006f940bd42d56f
I20260923 23:09:47.555168 27503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:47.555645 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:47.549753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55279 (local address 127.5.74.62:42529)
0923 23:09:47.549972 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:47.549983 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:47.550014 (+ 31us) server_negotiation.cc:407] Connection header received
0923 23:09:47.550114 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:47.550119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:47.550162 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:47.550275 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:47.550553 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:47.552009 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:47.553266 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:47.553859 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:47.553974 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:47.554795 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:47.554799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:47.554799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:47.555061 (+ 262us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:47.555101 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:47.555117 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:47.555189 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:47.555277 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 23:09:47.555444 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
I20260923 23:09:47.556641 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:47.564188 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:47.560635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34761 (local address 127.5.74.62:42529)
0923 23:09:47.560680 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:47.560683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:47.560758 (+ 75us) server_negotiation.cc:407] Connection header received
0923 23:09:47.560823 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:47.560824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:47.560836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:47.560898 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:47.561062 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:47.561829 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:47.562833 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:47.563144 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:47.563219 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:47.563898 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:47.563899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:47.563899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:47.563905 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:47.563921 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:47.563926 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:47.563986 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:47.564047 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 23:09:47.564078 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:09:47.579768 27503 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:47.580695 27503 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:47.587404 27503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3704aa69f0f749d583186b6e0c2de525: Generated new TSK 0
I20260923 23:09:47.587646 27503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:47.593007 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 27417
2026-09-23T23:09:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:47 dist-test-slave-j4c5 krb5kdc[27516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:47 dist-test-slave-j4c5 krb5kdc[27516](info): set up 2 sockets
Sep 23 23:09:47 dist-test-slave-j4c5 krb5kdc[27516](info): commencing operation
krb5kdc: starting...
W20260923 23:09:48.668524 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:48 dist-test-slave-j4c5 krb5kdc[27516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:48.796756 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:39651
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:39651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:48.883490 27529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:48.883735 27529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:48.883762 27529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:48.885087 27529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:48.885145 27529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:48.885160 27529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:48.885174 27529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:48.885186 27529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:48.886893 27529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:39651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:39651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:48.887204 27529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:48.887471 27529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:48.892745 27534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:48.892730 27537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:48.892730 27535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:48.893579 27529 server_base.cc:1034] running on GCE node
I20260923 23:09:48.893963 27529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:48.894259 27529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:48.895442 27529 hybrid_clock.cc:648] HybridClock initialized: now 1790204988895425 us; error 33 us; skew 500 ppm
Sep 23 23:09:48 dist-test-slave-j4c5 krb5kdc[27516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204988, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:48.906978 27529 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:48.907552 27529 webserver.cc:460] Webserver started at http://127.5.74.62:45051/ using document root <none> and password file <none>
I20260923 23:09:48.907814 27529 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:48.907862 27529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:48.908011 27529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:48.909204 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "cebe078e9243476d9b64cd580538b4e6"
format_stamp: "Formatted at 2026-09-23 23:09:48 on dist-test-slave-j4c5"
server_key: "5d2a12e237633ce6427ab8361a67fd1e"
server_key_iv: "a689c780e6969401eec9ab645fc307bf"
server_key_version: "encryptionkey@0"
I20260923 23:09:48.909572 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "cebe078e9243476d9b64cd580538b4e6"
format_stamp: "Formatted at 2026-09-23 23:09:48 on dist-test-slave-j4c5"
server_key: "5d2a12e237633ce6427ab8361a67fd1e"
server_key_iv: "a689c780e6969401eec9ab645fc307bf"
server_key_version: "encryptionkey@0"
I20260923 23:09:48.911088 27529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:09:48.912267 27544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:48.912776 27529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:48.912931 27529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "cebe078e9243476d9b64cd580538b4e6"
format_stamp: "Formatted at 2026-09-23 23:09:48 on dist-test-slave-j4c5"
server_key: "5d2a12e237633ce6427ab8361a67fd1e"
server_key_iv: "a689c780e6969401eec9ab645fc307bf"
server_key_version: "encryptionkey@0"
I20260923 23:09:48.913025 27529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:48.916817 27529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:48.919298 27529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:48.919520 27529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:48.924348 27529 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:39651
I20260923 23:09:48.924582 27596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:39651 every 8 connection(s)
I20260923 23:09:48.924999 27529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:48.926131 27597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:48.929049 27597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6: Bootstrap starting.
I20260923 23:09:48.929678 27597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:48.929975 27597 log.cc:824] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:48.930756 27597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6: No bootstrap required, opened a new log
I20260923 23:09:48.932279 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 27529
I20260923 23:09:48.932413 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:48.932607 5416 external_mini_cluster.cc:1428] Setting key 770038c81d4916cc6850921c304dd734
I20260923 23:09:48.932765 27597 raft_consensus.cc:348] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebe078e9243476d9b64cd580538b4e6" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39651 } }
I20260923 23:09:48.932951 27597 raft_consensus.cc:374] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:09:48.932996 27597 raft_consensus.cc:729] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebe078e9243476d9b64cd580538b4e6, State: Initialized, Role: FOLLOWER
I20260923 23:09:48.933182 27597 consensus_queue.cc:260] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebe078e9243476d9b64cd580538b4e6" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39651 } }
I20260923 23:09:48.933285 27597 raft_consensus.cc:388] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:09:48.933346 27597 raft_consensus.cc:482] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:09:48.933437 27597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:48.934520 27597 raft_consensus.cc:504] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebe078e9243476d9b64cd580538b4e6" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39651 } }
I20260923 23:09:48.934650 27597 leader_election.cc:302] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cebe078e9243476d9b64cd580538b4e6; no voters:
I20260923 23:09:48.934839 27597 leader_election.cc:288] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:09:48.934862 27602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:48.935026 27602 raft_consensus.cc:686] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [term 1 LEADER]: Becoming Leader. State: Replica: cebe078e9243476d9b64cd580538b4e6, State: Running, Role: LEADER
I20260923 23:09:48.935166 27597 sys_catalog.cc:564] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:48.935091 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:48.933174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38403 (local address 127.5.74.62:39651)
0923 23:09:48.933455 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:48.933465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:48.933506 (+ 41us) server_negotiation.cc:407] Connection header received
0923 23:09:48.933661 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:48.933667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:48.933690 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:48.933836 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:48.934579 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:48.934583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:48.934584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:48.934588 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:48.934658 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:48.934674 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:48.934740 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:48.934812 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 23:09:48.934857 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":133,"threads_started":1}
I20260923 23:09:48.935338 27602 consensus_queue.cc:237] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebe078e9243476d9b64cd580538b4e6" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39651 } }
I20260923 23:09:48.935705 27604 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cebe078e9243476d9b64cd580538b4e6. Latest consensus state: current_term: 1 leader_uuid: "cebe078e9243476d9b64cd580538b4e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebe078e9243476d9b64cd580538b4e6" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39651 } } }
I20260923 23:09:48.935710 27603 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cebe078e9243476d9b64cd580538b4e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebe078e9243476d9b64cd580538b4e6" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 39651 } } }
I20260923 23:09:48.935868 27604 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:48.936123 27603 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:48.936182 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 23:09:48.937245 27617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:48.937315 27617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:48.937381 27611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:48.937623 27611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:48.939196 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:48.937963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59825 (local address 127.5.74.62:39651)
0923 23:09:48.938215 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:48.938217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:48.938226 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:09:48.938242 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:48.938243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:48.938246 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:48.938302 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:48.938960 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:48.938961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:48.938962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:09:48.938966 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:48.938980 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:09:48.938985 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:48.939031 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:48.939094 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 23:09:48.939109 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260923 23:09:48.939296 27611 catalog_manager.cc:1347] Generated new cluster ID: b30a3f23b2ff49f987d2cf24d44f644f
I20260923 23:09:48.939357 27611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:48.952378 27611 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:48.952647 27611 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:48.957803 27611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cebe078e9243476d9b64cd580538b4e6: Generated new TSK 0
I20260923 23:09:48.958101 27611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:09:48.964778 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 27529
2026-09-23T23:09:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1372 ms)
[----------] 18 tests from AuthTokenIssuingTest (24976 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:49 dist-test-slave-j4c5 krb5kdc[27628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:49 dist-test-slave-j4c5 krb5kdc[27628](info): set up 2 sockets
Sep 23 23:09:49 dist-test-slave-j4c5 krb5kdc[27628](info): commencing operation
krb5kdc: starting...
W20260923 23:09:50.032438 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:50.164609 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:33497
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:33497,127.5.74.61:42629,127.5.74.60:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:50.249063 27641 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:09:50.249356 27641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:50.249374 27641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:50.249397 27641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:50.250594 27641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:50.250649 27641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:50.250663 27641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:50.250676 27641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:50.250687 27641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:50.252270 27641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:33497,127.5.74.61:42629,127.5.74.60:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:33497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:50.252529 27641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:50.252789 27641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:50.258314 27649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:50.258318 27646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:50.258369 27641 server_base.cc:1034] running on GCE node
W20260923 23:09:50.258314 27647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:50.258884 27641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:50.259220 27641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:50.260406 27641 hybrid_clock.cc:648] HybridClock initialized: now 1790204990260384 us; error 40 us; skew 500 ppm
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:50.262976 27641 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:50.263486 27641 webserver.cc:460] Webserver started at http://127.5.74.62:38473/ using document root <none> and password file <none>
I20260923 23:09:50.263721 27641 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:50.263792 27641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:50.263943 27641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:50.265154 27641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "29ce4e9ead17468099ce25e3457caec5"
format_stamp: "Formatted at 2026-09-23 23:09:50 on dist-test-slave-j4c5"
server_key: "074d1bf54c3e21fd2d813c450737a466"
server_key_iv: "00f15155832e16592828e19e4afd6c95"
server_key_version: "encryptionkey@0"
I20260923 23:09:50.265527 27641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "29ce4e9ead17468099ce25e3457caec5"
format_stamp: "Formatted at 2026-09-23 23:09:50 on dist-test-slave-j4c5"
server_key: "074d1bf54c3e21fd2d813c450737a466"
server_key_iv: "00f15155832e16592828e19e4afd6c95"
server_key_version: "encryptionkey@0"
I20260923 23:09:50.267037 27641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:09:50.267999 27656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:50.268252 27641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:50.268360 27641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "29ce4e9ead17468099ce25e3457caec5"
format_stamp: "Formatted at 2026-09-23 23:09:50 on dist-test-slave-j4c5"
server_key: "074d1bf54c3e21fd2d813c450737a466"
server_key_iv: "00f15155832e16592828e19e4afd6c95"
server_key_version: "encryptionkey@0"
I20260923 23:09:50.268502 27641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:50.283815 27641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:50.286067 27641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:50.286254 27641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:50.291067 27641 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:33497
I20260923 23:09:50.291106 27708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:33497 every 8 connection(s)
I20260923 23:09:50.291497 27641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:50.292558 27709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:50.294153 27709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:50.300243 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 27641
I20260923 23:09:50.300370 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:50.300721 5416 external_mini_cluster.cc:1428] Setting key 2d6731df66140bd707ab166f2d1d8e4c
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:50.305794 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.295045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.5.74.62:33497)
0923 23:09:50.295190 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.295196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.295236 (+ 40us) server_negotiation.cc:407] Connection header received
0923 23:09:50.295735 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.295738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.295756 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.295841 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.296609 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.297515 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.298505 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.298826 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.298928 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.301636 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.301646 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.301646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.301655 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.303665 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.304301 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.304305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.304305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.304450 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.304737 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.304738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.304739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.305063 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.305143 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.305442 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.305618 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260923 23:09:50.305851 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.294864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:33497 (local address 127.0.0.1:41846)
0923 23:09:50.295079 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.295118 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:50.295586 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:50.295914 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:50.295917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:50.296316 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:50.296494 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.296498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.297651 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.297652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.298393 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.298395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.298961 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.298962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.299033 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.299676 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:50.299685 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:50.301533 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:50.303788 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.303796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.303799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.304123 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.304532 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.304533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.304533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.304658 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.305152 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:50.305159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:50.305251 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 23:09:50.305355 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 23:09:50.305549 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260923 23:09:50.306689 27709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:50.307101 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.306952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:42629 (local address 127.0.0.1:37326)
0923 23:09:50.307017 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.307074 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 23:09:50.307260 27659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
I20260923 23:09:50.307847 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.307725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:42629 (local address 127.0.0.1:37338)
0923 23:09:50.307785 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.307819 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 23:09:50.308040 27709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.61:42629: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.74.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab.
Entry for principal kudu/127.5.74.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab.
I20260923 23:09:50.332439 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.61:42629
--webserver_interface=127.5.74.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab
--principal=kudu/127.5.74.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:33497,127.5.74.61:42629,127.5.74.60:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:09:50.357321 27709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } attempt: 1
I20260923 23:09:50.357892 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.357733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:42629 (local address 127.0.0.1:37346)
0923 23:09:50.357811 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.357861 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 23:09:50.358295 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.358202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:42629 (local address 127.0.0.1:37354)
0923 23:09:50.358247 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.358275 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 23:09:50.358438 27709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.61:42629: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
W20260923 23:09:50.419757 27716 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:09:50.419981 27716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:50.420001 27716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:50.420024 27716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:50.421139 27716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:50.421177 27716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:50.421191 27716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:50.421204 27716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:50.421216 27716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:50.422650 27716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:33497,127.5.74.61:42629,127.5.74.60:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab
--principal=kudu/127.5.74.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.61:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.74.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:50.422859 27716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:50.423033 27716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:09:50.426594 27709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } attempt: 2
I20260923 23:09:50.427085 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.426929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:42629 (local address 127.0.0.1:37370)
0923 23:09:50.427005 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.427055 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 23:09:50.427543 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.427436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:42629 (local address 127.0.0.1:37372)
0923 23:09:50.427480 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.427510 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260923 23:09:50.427682 27709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.61:42629: Network error: Client connection negotiation failed: client connection to 127.5.74.61:42629: connect: Connection refused (error 111)
W20260923 23:09:50.428021 27724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:50.428089 27722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:50.428203 27721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:50.428503 27716 server_base.cc:1034] running on GCE node
I20260923 23:09:50.428763 27716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:50.429055 27716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:50.430260 27716 hybrid_clock.cc:648] HybridClock initialized: now 1790204990430243 us; error 28 us; skew 500 ppm
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:50.433020 27716 init.cc:377] Logged in from keytab as kudu/127.5.74.61@KRBTEST.COM (short username kudu)
I20260923 23:09:50.433480 27716 webserver.cc:460] Webserver started at http://127.5.74.61:37515/ using document root <none> and password file <none>
I20260923 23:09:50.433717 27716 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:50.433763 27716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:50.433908 27716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:50.435019 27716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/data/instance:
uuid: "746c82401ddf4b659505b7ab3cde66df"
format_stamp: "Formatted at 2026-09-23 23:09:50 on dist-test-slave-j4c5"
server_key: "22be75ab61e81840960153ab41d156f8"
server_key_iv: "ef96793cbbeb800e883d9a0c257827d5"
server_key_version: "encryptionkey@0"
I20260923 23:09:50.435381 27716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/wal/instance:
uuid: "746c82401ddf4b659505b7ab3cde66df"
format_stamp: "Formatted at 2026-09-23 23:09:50 on dist-test-slave-j4c5"
server_key: "22be75ab61e81840960153ab41d156f8"
server_key_iv: "ef96793cbbeb800e883d9a0c257827d5"
server_key_version: "encryptionkey@0"
I20260923 23:09:50.436914 27716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:09:50.437760 27731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:50.438005 27716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:09:50.438082 27716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/wal
uuid: "746c82401ddf4b659505b7ab3cde66df"
format_stamp: "Formatted at 2026-09-23 23:09:50 on dist-test-slave-j4c5"
server_key: "22be75ab61e81840960153ab41d156f8"
server_key_iv: "ef96793cbbeb800e883d9a0c257827d5"
server_key_version: "encryptionkey@0"
I20260923 23:09:50.438175 27716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:50.454283 27716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:50.456782 27716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:50.456955 27716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:50.460829 27716 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.61:42629
I20260923 23:09:50.460876 27783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.61:42629 every 8 connection(s)
I20260923 23:09:50.461247 27716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
I20260923 23:09:50.462282 27784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:50.463955 27784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:50.467994 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 27716
I20260923 23:09:50.468081 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-1/wal/instance
I20260923 23:09:50.468317 5416 external_mini_cluster.cc:1428] Setting key 08945f814bc2326abc2b79816bfb7cd2
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:50.474438 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.464901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.5.74.62:33497)
0923 23:09:50.464998 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.465000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.465127 (+ 127us) server_negotiation.cc:407] Connection header received
0923 23:09:50.465661 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.465662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.465675 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.465731 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.466482 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.467060 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.468270 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.468603 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.468681 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.471502 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.471504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.471504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.471513 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.473021 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.473435 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.473437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.473437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.473504 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.473769 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.473770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.473770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.473897 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.473974 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.474252 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.474336 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 23:09:50.474637 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.464790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:33497 (local address 127.0.0.1:41856)
0923 23:09:50.464991 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.465032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:50.465560 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:50.465811 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:50.465814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:50.466077 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:50.466281 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.466285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.467175 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.467176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.468127 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.468130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.468757 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.468758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.469009 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.469616 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:50.469625 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:50.471331 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:50.473123 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.473130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.473133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.473356 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.473575 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.473576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.473577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.473692 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.473965 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:50.473973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:50.474110 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 23:09:50.474219 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 23:09:50.474395 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260923 23:09:50.475247 27784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.74.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.60@KRBTEST.COM" created.
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
I20260923 23:09:50.483899 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.475573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:42629 (local address 127.0.0.1:37376)
0923 23:09:50.475681 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.475696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:50.475753 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:50.476043 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:50.476045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:50.476271 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:50.476342 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.476344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.477245 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.477246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.477889 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.477891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.478408 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.478409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.478470 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.478962 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:50.478968 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:50.480662 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:50.482513 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.482514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.482515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.482756 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.483007 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.483008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.483008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.483115 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.483518 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:50.483519 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:50.483564 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 23:09:50.483642 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 23:09:50.483775 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 23:09:50.483999 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.475614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.5.74.61:42629)
0923 23:09:50.475863 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.475867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.475887 (+ 20us) server_negotiation.cc:407] Connection header received
0923 23:09:50.475922 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.475924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.475936 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.476010 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.476421 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.477163 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.477977 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.478326 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.478389 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.480745 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.480756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.480757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.480765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.482419 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.482847 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.482850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.482851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.482913 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.483197 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.483198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.483199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.483411 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.483512 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.483675 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.483835 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":211,"thread_start_us":180,"threads_started":1}
I20260923 23:09:50.484644 27784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:50.485235 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.484913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46838)
0923 23:09:50.485129 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.485193 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
W20260923 23:09:50.485378 27734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
I20260923 23:09:50.485827 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.485730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46846)
0923 23:09:50.485780 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.485805 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:09:50.485932 27784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:39461: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab.
Entry for principal kudu/127.5.74.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab.
I20260923 23:09:50.499495 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.60:39461
--webserver_interface=127.5.74.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab
--principal=kudu/127.5.74.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:33497,127.5.74.61:42629,127.5.74.60:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:09:50.518893 27709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } attempt: 3
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
I20260923 23:09:50.526855 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.519445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.5.74.61:42629)
0923 23:09:50.519516 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.519518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.519530 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:09:50.519668 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.519669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.519684 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.519768 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.520259 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.520765 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.521518 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.521764 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.521845 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.524093 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.524095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.524096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.524105 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.525710 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.526124 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.526125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.526126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.526180 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.526381 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.526382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.526382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.526494 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.526570 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.526719 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.526785 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 23:09:50.526892 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.519372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:42629 (local address 127.0.0.1:37392)
0923 23:09:50.519440 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.519461 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:50.519583 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:50.519776 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:50.519777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:50.519956 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:50.520030 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.520032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.520860 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.520862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.521407 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.521410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.521842 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.521843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.521933 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.522458 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:50.522479 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:50.523838 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:50.525806 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.525807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.525808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.526045 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.526254 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.526255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.526255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.526322 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.526560 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:50.526561 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:50.526616 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 23:09:50.526686 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 23:09:50.526796 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 23:09:50.527315 27709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:50.527717 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.527602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46860)
0923 23:09:50.527657 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.527693 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 23:09:50.528054 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.527959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46874)
0923 23:09:50.528003 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.528037 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 23:09:50.528179 27709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:39461: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
I20260923 23:09:50.535182 27784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } attempt: 1
I20260923 23:09:50.535717 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.535579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46890)
0923 23:09:50.535641 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.535680 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 23:09:50.536090 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.535983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46894)
0923 23:09:50.536031 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.536058 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 23:09:50.536198 27784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:39461: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
I20260923 23:09:50.559361 27709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } attempt: 1
I20260923 23:09:50.559958 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.559793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46906)
0923 23:09:50.559867 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.559920 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 23:09:50.560437 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.560334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46912)
0923 23:09:50.560383 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.560415 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:09:50.560562 27709 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:39461: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
W20260923 23:09:50.586699 27791 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:09:50.586915 27791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:50.586931 27791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:50.586953 27791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:50.588141 27791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:50.588207 27791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:50.588222 27791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:50.588234 27791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:50.588245 27791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:50.589844 27791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:33497,127.5.74.61:42629,127.5.74.60:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab
--principal=kudu/127.5.74.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.60:39461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.74.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:50.590132 27791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:50.590373 27791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:50.595436 27796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:50.595461 27797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:50.595465 27799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:50.595781 27791 server_base.cc:1034] running on GCE node
I20260923 23:09:50.596091 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:50.596426 27791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:50.597590 27791 hybrid_clock.cc:648] HybridClock initialized: now 1790204990597576 us; error 28 us; skew 500 ppm
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:50.600127 27791 init.cc:377] Logged in from keytab as kudu/127.5.74.60@KRBTEST.COM (short username kudu)
I20260923 23:09:50.600612 27791 webserver.cc:460] Webserver started at http://127.5.74.60:35457/ using document root <none> and password file <none>
I20260923 23:09:50.600841 27791 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:50.600888 27791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:50.601043 27791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:50.602195 27791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/data/instance:
uuid: "7dc4755ca9ca42df9e944f4088afc901"
format_stamp: "Formatted at 2026-09-23 23:09:50 on dist-test-slave-j4c5"
server_key: "da3eda6b1f3819c924ab8c5b7ff4afa0"
server_key_iv: "9f0df72df79646257ee0605ae1bc0092"
server_key_version: "encryptionkey@0"
I20260923 23:09:50.602571 27791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/wal/instance:
uuid: "7dc4755ca9ca42df9e944f4088afc901"
format_stamp: "Formatted at 2026-09-23 23:09:50 on dist-test-slave-j4c5"
server_key: "da3eda6b1f3819c924ab8c5b7ff4afa0"
server_key_iv: "9f0df72df79646257ee0605ae1bc0092"
server_key_version: "encryptionkey@0"
I20260923 23:09:50.604023 27791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:50.604334 27784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } attempt: 2
I20260923 23:09:50.604828 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.604706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46924)
0923 23:09:50.604766 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.604802 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 23:09:50.605114 27806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:50.605253 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.605144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46932)
0923 23:09:50.605194 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.605224 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 23:09:50.605286 27791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
W20260923 23:09:50.605350 27784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:39461: Network error: Client connection negotiation failed: client connection to 127.5.74.60:39461: connect: Connection refused (error 111)
I20260923 23:09:50.605370 27791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/wal
uuid: "7dc4755ca9ca42df9e944f4088afc901"
format_stamp: "Formatted at 2026-09-23 23:09:50 on dist-test-slave-j4c5"
server_key: "da3eda6b1f3819c924ab8c5b7ff4afa0"
server_key_iv: "9f0df72df79646257ee0605ae1bc0092"
server_key_version: "encryptionkey@0"
I20260923 23:09:50.605445 27791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:50.628051 27791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:50.630467 27791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:50.630590 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:50.635219 27791 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.60:39461
I20260923 23:09:50.635253 27858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.60:39461 every 8 connection(s)
I20260923 23:09:50.635641 27791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
I20260923 23:09:50.635732 27709 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } attempt: 2
I20260923 23:09:50.637158 27859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:50.639238 27859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:50.645313 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 27791
I20260923 23:09:50.645428 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204714633713-5416-0/minicluster-data/master-2/wal/instance
I20260923 23:09:50.645637 5416 external_mini_cluster.cc:1428] Setting key f014f041351233e30e81a67155de858a
I20260923 23:09:50.647455 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.636107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46944)
0923 23:09:50.636204 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.636216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:50.636273 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:50.637408 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:50.637409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:50.637566 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:50.637614 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.637616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.639190 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.639192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.640126 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.640130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.640853 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.640855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.640922 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.641477 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:50.641484 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:50.643133 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:50.645048 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.645049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.645050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.645283 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.646168 (+ 885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.646169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.646170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.646242 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.647076 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:50.647077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:50.647131 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 23:09:50.647263 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 23:09:50.647357 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 23:09:50.648087 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.636265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.5.74.60:39461)
0923 23:09:50.636441 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.636450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.636488 (+ 38us) server_negotiation.cc:407] Connection header received
0923 23:09:50.637126 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.637135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.637209 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.637417 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.637724 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.639105 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.640240 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.640767 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.640848 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.643219 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.643226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.643227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.643238 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.644939 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.645370 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.645373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.645374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.645594 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.646578 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.646580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.646580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.646985 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.647078 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.647720 (+ 642us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.647848 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260923 23:09:50.648360 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.640082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.5.74.62:33497)
0923 23:09:50.640167 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.640169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.640357 (+ 188us) server_negotiation.cc:407] Connection header received
0923 23:09:50.640384 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.640385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.640394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.640461 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.640971 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.641516 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.642555 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.642871 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.642982 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.645431 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.645433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.645433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.645439 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.647071 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.647467 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.647468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.647468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.647551 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.647752 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.647753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.647753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.647872 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.647908 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.648202 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.648282 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:09:50.648414 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.639990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:33497 (local address 127.0.0.1:41864)
0923 23:09:50.640230 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.640282 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:50.640349 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:50.640491 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:50.640493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:50.640766 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:50.640872 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.640874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.641604 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.641605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.642467 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.642470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.643018 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.643020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.643104 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.643576 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:50.643598 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:50.645338 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:50.647139 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.647143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.647146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.647379 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.647631 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.647632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.647632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.647688 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.647946 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:50.647952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:50.648043 (+ 91us) client_negotiation.cc:769] Sending connection context
0923 23:09:50.648132 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 23:09:50.648297 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260923 23:09:50.649012 27859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:50.649935 27709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5: Bootstrap starting.
I20260923 23:09:50.650993 27709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:50.651329 27709 log.cc:824] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:50.652338 27709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5: No bootstrap required, opened a new log
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
I20260923 23:09:50.654641 27709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } }
I20260923 23:09:50.654794 27709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ce4e9ead17468099ce25e3457caec5, State: Initialized, Role: FOLLOWER
I20260923 23:09:50.654906 27709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } }
I20260923 23:09:50.655228 27866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.655424 27866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:50.655293 27709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:50.656788 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.649279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:42629 (local address 127.0.0.1:37402)
0923 23:09:50.649327 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.649337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:50.649372 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:50.649572 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:50.649573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:50.649702 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:50.649765 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.649767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.650566 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.650567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.651275 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.651277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.651749 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.651750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.651822 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.652312 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:50.652318 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:50.653588 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:50.655463 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.655464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.655465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.655787 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.656054 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.656055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.656055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.656124 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.656472 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:50.656473 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:50.656518 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 23:09:50.656593 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 23:09:50.656684 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 23:09:50.656828 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.649340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.5.74.61:42629)
0923 23:09:50.649393 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.649396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.649407 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:09:50.649434 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.649435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.649446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.649516 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.649921 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.650472 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.651413 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.651661 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.651740 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.653676 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.653677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.653678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.653685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.655369 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.655914 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.655915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.655916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.655982 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.656227 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.656228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.656229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.656385 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.656447 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.656639 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.656726 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:09:50.657248 27859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:50.658082 27864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.646044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.5.74.60:39461)
0923 23:09:50.646352 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.646354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.646360 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:09:50.646379 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.646379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.646396 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.646467 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.647241 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.647937 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.649889 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.650151 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.650211 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.653190 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.653192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.653193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.653200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.655777 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.656330 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.656331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.656331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.656381 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.656583 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.656583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.656584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.656693 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.657725 (+ 1032us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.657768 (+ 43us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.657925 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":85,"threads_started":1}
W20260923 23:09:50.660557 27877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:50.660621 27877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
I20260923 23:09:50.658757 27641 master_runner.cc:186] Error getting master registration for 127.5.74.61:42629: OK, instance_id { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" instance_seqno: 1790204990459721 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:09:50.661866 27641 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:50.665194 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.657574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46954 (local address 127.5.74.60:39461)
0923 23:09:50.657726 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.657728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.657734 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:09:50.657748 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.657748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.657756 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.657797 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.658178 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.658892 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.659566 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.659874 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.659933 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.662090 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.662091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.662091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.662098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.663669 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.664064 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.664064 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.664065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.664114 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.664312 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.664313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.664313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.664429 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.664497 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.664945 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.665100 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":138}
I20260923 23:09:50.665403 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.657567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46954)
0923 23:09:50.657613 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.657624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:50.657666 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:50.657829 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:50.657830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:50.658038 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:50.658077 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.658079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.658970 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.658971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.659464 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.659466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.659974 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.659974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.660018 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.660575 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:50.660581 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:50.661997 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:50.663758 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.663759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.663760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.663974 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.664196 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.664197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.664197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.664251 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.664765 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:50.664766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:50.664815 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 23:09:50.664900 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 23:09:50.665014 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 23:09:50.667101 27859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901: Bootstrap starting.
I20260923 23:09:50.667227 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.658761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.5.74.62:33497)
0923 23:09:50.658795 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.658797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.658807 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:09:50.658867 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.658867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.658876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.658942 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.659538 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.660254 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.661144 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.661367 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.661451 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.664281 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.664282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.664283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.664290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.666091 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.666438 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.666440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.666440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.666496 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.666712 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.666713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.666713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.666835 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.666985 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.667052 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.667130 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:09:50.667881 27859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:50.668219 27859 log.cc:824] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:50.668982 27859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901: No bootstrap required, opened a new log
I20260923 23:09:50.671033 27859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } }
I20260923 23:09:50.671211 27859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc4755ca9ca42df9e944f4088afc901, State: Initialized, Role: FOLLOWER
I20260923 23:09:50.671303 27859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } }
I20260923 23:09:50.671623 27859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:50.671594 27880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.671985 27880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [sys.catalog]: This master's current role is: FOLLOWER
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
W20260923 23:09:50.674152 27891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:50.674252 27891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:50.675972 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.667877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.5.74.61:42629)
0923 23:09:50.667917 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.667919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.667927 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:09:50.667973 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.667974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.667983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.668038 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.668703 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.669244 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.669906 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.670128 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.670206 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.673043 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.673044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.673045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.673053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.674766 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.675183 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.675188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.675189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.675241 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.675451 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.675452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.675452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.675571 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.675747 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.675784 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.675861 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:09:50.677196 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:50.675177 27791 master_runner.cc:186] Error getting master registration for 127.5.74.61:42629: OK, instance_id { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" instance_seqno: 1790204990459721 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:09:50.677449 27791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:09:50.678118 27683 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "29ce4e9ead17468099ce25e3457caec5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41878
I20260923 23:09:50.678244 27683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 23:09:50.678306 27683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:50.679075 27683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } }
I20260923 23:09:50.679323 27683 leader_election.cc:288] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [CANDIDATE]: Term 1 election: Requested vote from peers 746c82401ddf4b659505b7ab3cde66df (127.5.74.61:42629), 7dc4755ca9ca42df9e944f4088afc901 (127.5.74.60:39461)
I20260923 23:09:50.679590 27833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29ce4e9ead17468099ce25e3457caec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7dc4755ca9ca42df9e944f4088afc901"
I20260923 23:09:50.679723 27833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:50.680784 27833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29ce4e9ead17468099ce25e3457caec5 in term 1.
I20260923 23:09:50.681082 27659 leader_election.cc:302] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ce4e9ead17468099ce25e3457caec5, 7dc4755ca9ca42df9e944f4088afc901; no voters:
I20260923 23:09:50.681236 27866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:50.681407 27866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [term 1 LEADER]: Becoming Leader. State: Replica: 29ce4e9ead17468099ce25e3457caec5, State: Running, Role: LEADER
I20260923 23:09:50.681525 27866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } }
I20260923 23:09:50.683048 27900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ce4e9ead17468099ce25e3457caec5. Latest consensus state: current_term: 1 leader_uuid: "29ce4e9ead17468099ce25e3457caec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.683146 27900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:50.683364 27902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:50.683583 27902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:50.685285 27833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [term 1 FOLLOWER]: Refusing update from remote peer 29ce4e9ead17468099ce25e3457caec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:50.685781 27900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:50.687451 27880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ce4e9ead17468099ce25e3457caec5. Latest consensus state: current_term: 1 leader_uuid: "29ce4e9ead17468099ce25e3457caec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.687556 27880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:50.691890 27900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29ce4e9ead17468099ce25e3457caec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.691993 27900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:50.692624 27903 mvcc.cc:205] Tried to move back new op lower bound from 7332679641844408320 to 7332679641832677376. Current Snapshot: MvccSnapshot[applied={T|T < 7332679641844408320}]
I20260923 23:09:50.693857 27880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29ce4e9ead17468099ce25e3457caec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.693980 27880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:50.694157 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.680552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.5.74.62:33497)
0923 23:09:50.680603 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.680605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.680792 (+ 187us) server_negotiation.cc:407] Connection header received
0923 23:09:50.680917 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.680918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.680929 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.680962 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.681930 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.682677 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.683670 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.683978 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.684046 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.689768 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.689770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.689770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.689778 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.692026 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.693232 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.693233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.693234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.693296 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.693527 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.693528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.693528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.693658 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.693731 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.693973 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.694062 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:09:50.694236 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.680951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.5.74.61:42629)
0923 23:09:50.681807 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.681809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.682225 (+ 416us) server_negotiation.cc:407] Connection header received
0923 23:09:50.682600 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.682601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.682613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.682663 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.686602 (+ 3939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.687244 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.688273 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.688586 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.688634 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.690500 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.690501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.690502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.690520 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.692188 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.692746 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.692747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.692747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.692802 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.693080 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.693081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.693082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.693216 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.693749 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.694066 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.694147 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831}
I20260923 23:09:50.694943 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.681128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.5.74.60:39461)
0923 23:09:50.681166 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.681168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.682320 (+ 1152us) server_negotiation.cc:407] Connection header received
0923 23:09:50.682489 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.682490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.682501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.682530 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.684094 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.684799 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.685731 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.686006 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.686077 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.687646 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.687647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.687648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.687654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.689324 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.689820 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.689822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.689822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.689886 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.692271 (+ 2385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.692272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.692273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.692413 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.694731 (+ 2318us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.694773 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.694861 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:09:50.695384 27904 mvcc.cc:205] Tried to move back new op lower bound from 7332679641844408320 to 7332679641832677376. Current Snapshot: MvccSnapshot[applied={T|T < 7332679641844408320}]
I20260923 23:09:50.695806 27902 catalog_manager.cc:1347] Generated new cluster ID: a116294b4de64cf7bd1c7b18a41fb81c
I20260923 23:09:50.695865 27902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:50.696463 27784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } attempt: 3
Sep 23 23:09:50 dist-test-slave-j4c5 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
I20260923 23:09:50.705520 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.696683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:39461 (local address 127.0.0.1:46970)
0923 23:09:50.696756 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:50.696765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:50.696796 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:50.697027 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:50.697028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:50.697177 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:50.697215 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.697216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.698095 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.698096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.698869 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:50.698871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.699389 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.699389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:50.699452 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.700317 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:50.700322 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:50.701771 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:50.704429 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.704430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.704430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.704641 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.704841 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:50.704842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:50.704842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:50.704908 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:50.705201 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:50.705202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:50.705255 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 23:09:50.705312 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 23:09:50.705419 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 23:09:50.705951 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:50.696858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.5.74.60:39461)
0923 23:09:50.696901 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.696902 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.696909 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:09:50.696939 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.696940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.696948 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.696990 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.697291 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.698003 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.698973 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:50.699263 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:50.699316 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:50.701876 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:50.701878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:50.701878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:50.701884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:50.704335 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.704715 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.704716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.704716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.704775 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:50.704983 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:50.704984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:50.704984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:50.705116 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:50.705168 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:50.705737 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 23:09:50.705828 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:09:50.707398 27784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df: Bootstrap starting.
I20260923 23:09:50.708105 27784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:50.708467 27784 log.cc:824] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:50.709489 27784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df: No bootstrap required, opened a new log
I20260923 23:09:50.711645 27784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } }
I20260923 23:09:50.711799 27784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746c82401ddf4b659505b7ab3cde66df, State: Initialized, Role: FOLLOWER
I20260923 23:09:50.711881 27784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } }
I20260923 23:09:50.712261 27784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:50.712294 27758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29ce4e9ead17468099ce25e3457caec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "746c82401ddf4b659505b7ab3cde66df"
I20260923 23:09:50.712407 27758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:50.713197 27758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29ce4e9ead17468099ce25e3457caec5 in term 1.
I20260923 23:09:50.713675 27757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [term 1 FOLLOWER]: Refusing update from remote peer 29ce4e9ead17468099ce25e3457caec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:50.714156 27866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 23:09:50.714417 27920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:50.714474 27920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:50.716593 27923 mvcc.cc:205] Tried to move back new op lower bound from 7332679641844408320 to 7332679641832677376. Current Snapshot: MvccSnapshot[applied={T|T < 7332679641844408320}]
I20260923 23:09:50.717159 27925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ce4e9ead17468099ce25e3457caec5. Latest consensus state: current_term: 1 leader_uuid: "29ce4e9ead17468099ce25e3457caec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.717315 27925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:50.717393 27925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29ce4e9ead17468099ce25e3457caec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.717482 27925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:50.717968 27922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29ce4e9ead17468099ce25e3457caec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.718066 27922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:50.749396 27866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29ce4e9ead17468099ce25e3457caec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ce4e9ead17468099ce25e3457caec5" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 33497 } } peers { permanent_uuid: "746c82401ddf4b659505b7ab3cde66df" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 42629 } } peers { permanent_uuid: "7dc4755ca9ca42df9e944f4088afc901" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 39461 } } }
I20260923 23:09:50.749542 27866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5 [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:50.752244 27902 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:50.753033 27902 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:50.761199 27902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29ce4e9ead17468099ce25e3457caec5: Generated new TSK 0
I20260923 23:09:50.761426 27902 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:09:50.788630 27712 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.786956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.5.74.62:33497)
0923 23:09:50.787004 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.787007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.787874 (+ 867us) server_negotiation.cc:407] Connection header received
0923 23:09:50.788003 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.788004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.788015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.788047 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.788557 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41890: BlockingRecv error: recv got EOF from 127.0.0.1:41890 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 23:09:50.789664 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.787326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.5.74.61:42629)
0923 23:09:50.787366 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.787368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.788851 (+ 1483us) server_negotiation.cc:407] Connection header received
0923 23:09:50.789134 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.789135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.789148 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.789187 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.789597 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37434: BlockingRecv error: recv got EOF from 127.0.0.1:37434 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 23:09:50.790262 27861 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.787608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.5.74.60:39461)
0923 23:09:50.787648 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.787650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.788996 (+ 1346us) server_negotiation.cc:407] Connection header received
0923 23:09:50.789833 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.789835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.789847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.789878 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.790204 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46980: BlockingRecv error: recv got EOF from 127.0.0.1:46980 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 23:09:50.793088 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.792230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.5.74.61:42629)
0923 23:09:50.792268 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.792270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.792511 (+ 241us) server_negotiation.cc:407] Connection header received
0923 23:09:50.792647 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.792648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.792669 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.792700 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.793026 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37448: BlockingRecv error: recv got EOF from 127.0.0.1:37448 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:09:50.796186 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.795359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.5.74.61:42629)
0923 23:09:50.795393 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.795395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.795633 (+ 238us) server_negotiation.cc:407] Connection header received
0923 23:09:50.795725 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.795726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.795736 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.795765 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.796101 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37450: BlockingRecv error: recv got EOF from 127.0.0.1:37450 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 23:09:50.800101 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.799287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.5.74.61:42629)
0923 23:09:50.799330 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.799332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.799580 (+ 248us) server_negotiation.cc:407] Connection header received
0923 23:09:50.799672 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.799673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.799684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.799711 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.800041 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37464: BlockingRecv error: recv got EOF from 127.0.0.1:37464 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 23:09:50.806327 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.805372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.5.74.61:42629)
0923 23:09:50.805417 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.805419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.805707 (+ 288us) server_negotiation.cc:407] Connection header received
0923 23:09:50.805818 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.805819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.805831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.805863 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.806267 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37478: BlockingRecv error: recv got EOF from 127.0.0.1:37478 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 23:09:50.816766 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.815674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.5.74.61:42629)
0923 23:09:50.815724 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.815726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.816036 (+ 310us) server_negotiation.cc:407] Connection header received
0923 23:09:50.816258 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.816259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.816272 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.816306 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.816707 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37490: BlockingRecv error: recv got EOF from 127.0.0.1:37490 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 23:09:50.835203 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.834133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.5.74.61:42629)
0923 23:09:50.834180 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.834182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.834490 (+ 308us) server_negotiation.cc:407] Connection header received
0923 23:09:50.834601 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.834602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.834615 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.834648 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.835140 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37498: BlockingRecv error: recv got EOF from 127.0.0.1:37498 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 23:09:50.869923 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.868718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.5.74.61:42629)
0923 23:09:50.868768 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.868770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.869246 (+ 476us) server_negotiation.cc:407] Connection header received
0923 23:09:50.869363 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.869364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.869378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.869410 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.869830 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37502: BlockingRecv error: recv got EOF from 127.0.0.1:37502 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 23:09:50.936429 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:50.935625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.5.74.61:42629)
0923 23:09:50.935729 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:50.935731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:50.935803 (+ 72us) server_negotiation.cc:407] Connection header received
0923 23:09:50.935858 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:50.935859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:50.935869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:50.935939 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:50.936337 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37512: BlockingRecv error: recv got EOF from 127.0.0.1:37512 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 23:09:51.066540 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:51.065735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.5.74.61:42629)
0923 23:09:51.065809 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:51.065811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:51.065923 (+ 112us) server_negotiation.cc:407] Connection header received
0923 23:09:51.066031 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:51.066034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:51.066049 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:51.066117 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:51.066459 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37514: BlockingRecv error: recv got EOF from 127.0.0.1:37514 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 23:09:51.324838 27788 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:51.323937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.5.74.61:42629)
0923 23:09:51.324037 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:51.324039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:51.324051 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:09:51.324083 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:51.324084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:51.324094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:51.324184 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:51.324737 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37526: BlockingRecv error: recv got EOF from 127.0.0.1:37526 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 23:09:51.674747 27891 catalog_manager.cc:1259] Loaded cluster ID: a116294b4de64cf7bd1c7b18a41fb81c
I20260923 23:09:51.674860 27891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901: loading cluster ID for follower catalog manager: success
I20260923 23:09:51.675698 27891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901: acquiring CA information for follower catalog manager: success
I20260923 23:09:51.676004 27891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7dc4755ca9ca42df9e944f4088afc901: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:09:51.714913 27920 catalog_manager.cc:1259] Loaded cluster ID: a116294b4de64cf7bd1c7b18a41fb81c
I20260923 23:09:51.715009 27920 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df: loading cluster ID for follower catalog manager: success
I20260923 23:09:51.715878 27920 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df: acquiring CA information for follower catalog manager: success
I20260923 23:09:51.716245 27920 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 746c82401ddf4b659505b7ab3cde66df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:09:51.841797 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:51.838275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.5.74.61:42629)
0923 23:09:51.838436 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:51.838438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:51.838449 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:09:51.838487 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:51.838488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:51.838581 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:51.838665 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:51.838856 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.839635 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.840805 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.841069 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.841163 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:51.841372 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:51.841547 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:51.841640 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:51.841662 (+ 22us) server_negotiation.cc:299] Negotiation successful
0923 23:09:51.841685 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260923 23:09:51.847415 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:51.843978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.5.74.60:39461)
0923 23:09:51.844090 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:51.844092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:51.844102 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:09:51.844228 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:51.844230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:51.844280 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:51.844376 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:51.844502 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.845408 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.846373 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.846662 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.846745 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:51.846979 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:51.847123 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:51.847202 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:51.847297 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 23:09:51.847325 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260923 23:09:51.852982 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:51.849908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.5.74.60:39461)
0923 23:09:51.849962 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:51.849964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:51.850028 (+ 64us) server_negotiation.cc:407] Connection header received
0923 23:09:51.850126 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:51.850126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:51.850149 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:51.850199 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:51.850340 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.850796 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.852124 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.852450 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.852492 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:51.852700 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:51.852733 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:51.852763 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:51.852878 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 23:09:51.852902 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:09:51.852982 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:51.849709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.5.74.61:42629)
0923 23:09:51.849754 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:51.849756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:51.849799 (+ 43us) server_negotiation.cc:407] Connection header received
0923 23:09:51.849909 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:51.849910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:51.849922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:51.849952 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:51.850130 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.850878 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.852125 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.852450 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.852492 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:51.852700 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:51.852733 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:51.852763 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:51.852893 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 23:09:51.852915 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:09:51.853398 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:51.849513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.5.74.62:33497)
0923 23:09:51.849631 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:51.849633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:51.849645 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:09:51.849770 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:51.849771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:51.849825 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:51.849902 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:51.850097 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.850878 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.852180 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:51.852466 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:51.852533 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:51.852762 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:51.852880 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:51.852935 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:51.853285 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 23:09:51.853311 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260923 23:09:51.855502 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 27641
I20260923 23:09:51.861379 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 27716
I20260923 23:09:51.867400 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 27791
2026-09-23T23:09:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2906 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:51 dist-test-slave-j4c5 krb5kdc[28009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:51 dist-test-slave-j4c5 krb5kdc[28009](info): set up 2 sockets
Sep 23 23:09:51 dist-test-slave-j4c5 krb5kdc[28009](info): commencing operation
krb5kdc: starting...
W20260923 23:09:52.961144 5416 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/test-user.keytab.
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:09:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.74.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
Entry for principal kudu/127.5.74.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab.
I20260923 23:09:53.101603 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:45951
--webserver_interface=127.5.74.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:45951,127.5.74.61:45099,127.5.74.60:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:09:53.191238 28022 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:09:53.191490 28022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:53.191510 28022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:53.191534 28022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:53.192965 28022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:53.193032 28022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:53.193046 28022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:53.193066 28022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:53.193079 28022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:53.194785 28022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:45951,127.5.74.61:45099,127.5.74.60:40503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.62.keytab
--principal=kudu/127.5.74.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.62:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.74.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:53.195077 28022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:53.195367 28022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:53.200572 28027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:53.200608 28030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:53.200601 28028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:53.200878 28022 server_base.cc:1034] running on GCE node
I20260923 23:09:53.201175 28022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:53.201553 28022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:53.202749 28022 hybrid_clock.cc:648] HybridClock initialized: now 1790204993202719 us; error 40 us; skew 500 ppm
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:53.205593 28022 init.cc:377] Logged in from keytab as kudu/127.5.74.62@KRBTEST.COM (short username kudu)
I20260923 23:09:53.206084 28022 webserver.cc:460] Webserver started at http://127.5.74.62:41859/ using document root <none> and password file <none>
I20260923 23:09:53.206475 28022 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:53.206533 28022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:53.206812 28022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:53.208130 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/data/instance:
uuid: "7585c18053c34f07ab17656ba70f284b"
format_stamp: "Formatted at 2026-09-23 23:09:53 on dist-test-slave-j4c5"
server_key: "a68519c327113a224c68173da2aecfd1"
server_key_iv: "5ed89ff9090f20bc907e6dba6ce045f4"
server_key_version: "encryptionkey@0"
I20260923 23:09:53.208647 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/wal/instance:
uuid: "7585c18053c34f07ab17656ba70f284b"
format_stamp: "Formatted at 2026-09-23 23:09:53 on dist-test-slave-j4c5"
server_key: "a68519c327113a224c68173da2aecfd1"
server_key_iv: "5ed89ff9090f20bc907e6dba6ce045f4"
server_key_version: "encryptionkey@0"
I20260923 23:09:53.210182 28022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:09:53.211076 28037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:53.211321 28022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:53.211418 28022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/wal
uuid: "7585c18053c34f07ab17656ba70f284b"
format_stamp: "Formatted at 2026-09-23 23:09:53 on dist-test-slave-j4c5"
server_key: "a68519c327113a224c68173da2aecfd1"
server_key_iv: "5ed89ff9090f20bc907e6dba6ce045f4"
server_key_version: "encryptionkey@0"
I20260923 23:09:53.211480 28022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:53.232491 28022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:53.234880 28022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:53.235001 28022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:53.239480 28022 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.62:45951
I20260923 23:09:53.239566 28089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.62:45951 every 8 connection(s)
I20260923 23:09:53.239945 28022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/data/info.pb
I20260923 23:09:53.241173 28090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:53.242951 28090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:53.248039 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 28022
I20260923 23:09:53.248188 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-0/wal/instance
I20260923 23:09:53.248435 5416 external_mini_cluster.cc:1428] Setting key 8caf33e90d3b100866423d178884e5fb
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:53.255240 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.243636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:45951 (local address 127.0.0.1:40814)
0923 23:09:53.243845 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.243892 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:53.244521 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:53.244865 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:53.244868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:53.245140 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:53.245300 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.245304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.246356 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.246357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.247052 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.247054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.247627 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.247628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.247694 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.248398 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:53.248409 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:53.250200 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:53.253213 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.253220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.253223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.253438 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.253788 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.253789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.253789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.253880 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.254436 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:53.254442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:53.254539 (+ 97us) client_negotiation.cc:769] Sending connection context
0923 23:09:53.254639 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 23:09:53.254804 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":114,"threads_started":1}
I20260923 23:09:53.255252 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.243730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.5.74.62:45951)
0923 23:09:53.243933 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.243937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.244023 (+ 86us) server_negotiation.cc:407] Connection header received
0923 23:09:53.244691 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.244693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.244715 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.244809 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.245436 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.246254 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.247141 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.247521 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.247608 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.250308 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.250317 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.250318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.250327 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.253076 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.253576 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.253581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.253582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.253705 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.253956 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.253958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.253958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.254318 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.254536 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.254681 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.254888 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":138,"thread_start_us":112,"threads_started":1}
I20260923 23:09:53.256340 28090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:53.256752 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.256620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:45099 (local address 127.0.0.1:60118)
0923 23:09:53.256674 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.256722 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:09:53.256989 28038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
I20260923 23:09:53.257565 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.257451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:45099 (local address 127.0.0.1:60128)
0923 23:09:53.257509 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.257540 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 23:09:53.257687 28090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.61:45099: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.74.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab.
Entry for principal kudu/127.5.74.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab.
I20260923 23:09:53.281462 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.61:45099
--webserver_interface=127.5.74.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab
--principal=kudu/127.5.74.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:45951,127.5.74.61:45099,127.5.74.60:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:09:53.306937 28090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } attempt: 1
I20260923 23:09:53.307545 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.307388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:45099 (local address 127.0.0.1:60130)
0923 23:09:53.307466 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.307512 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 23:09:53.307919 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.307820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:45099 (local address 127.0.0.1:60136)
0923 23:09:53.307870 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.307895 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:09:53.308013 28090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.61:45099: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
W20260923 23:09:53.370515 28097 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:09:53.370796 28097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:53.370828 28097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:53.370875 28097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:53.372073 28097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:53.372155 28097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:53.372201 28097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:53.372236 28097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:53.372257 28097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:53.373906 28097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:45951,127.5.74.61:45099,127.5.74.60:40503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.61.keytab
--principal=kudu/127.5.74.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.61:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.74.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:53.374240 28097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:53.374495 28097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:09:53.376173 28090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } attempt: 2
I20260923 23:09:53.376679 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.376529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:45099 (local address 127.0.0.1:60146)
0923 23:09:53.376603 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.376650 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 23:09:53.377089 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.376954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:45099 (local address 127.0.0.1:60152)
0923 23:09:53.377016 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.377058 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:09:53.377266 28090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.61:45099: Network error: Client connection negotiation failed: client connection to 127.5.74.61:45099: connect: Connection refused (error 111)
W20260923 23:09:53.380299 28105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:53.380355 28103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:53.380437 28102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:53.380458 28097 server_base.cc:1034] running on GCE node
I20260923 23:09:53.380878 28097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:53.381218 28097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:53.382436 28097 hybrid_clock.cc:648] HybridClock initialized: now 1790204993382390 us; error 64 us; skew 500 ppm
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:53.385552 28097 init.cc:377] Logged in from keytab as kudu/127.5.74.61@KRBTEST.COM (short username kudu)
I20260923 23:09:53.386089 28097 webserver.cc:460] Webserver started at http://127.5.74.61:39341/ using document root <none> and password file <none>
I20260923 23:09:53.386312 28097 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:53.386359 28097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:53.386461 28097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:53.387750 28097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/data/instance:
uuid: "fbd3d9bd2bbf41a28853416625020c5a"
format_stamp: "Formatted at 2026-09-23 23:09:53 on dist-test-slave-j4c5"
server_key: "ae331fa968296debf1d78faec9194e2f"
server_key_iv: "7ae027e021b9ccc7360b3bd13bbb6e91"
server_key_version: "encryptionkey@0"
I20260923 23:09:53.388079 28097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/wal/instance:
uuid: "fbd3d9bd2bbf41a28853416625020c5a"
format_stamp: "Formatted at 2026-09-23 23:09:53 on dist-test-slave-j4c5"
server_key: "ae331fa968296debf1d78faec9194e2f"
server_key_iv: "7ae027e021b9ccc7360b3bd13bbb6e91"
server_key_version: "encryptionkey@0"
I20260923 23:09:53.389689 28097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:09:53.390586 28112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:53.390828 28097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:09:53.390926 28097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/wal
uuid: "fbd3d9bd2bbf41a28853416625020c5a"
format_stamp: "Formatted at 2026-09-23 23:09:53 on dist-test-slave-j4c5"
server_key: "ae331fa968296debf1d78faec9194e2f"
server_key_iv: "7ae027e021b9ccc7360b3bd13bbb6e91"
server_key_version: "encryptionkey@0"
I20260923 23:09:53.391036 28097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:53.399045 28097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:53.401649 28097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:53.401777 28097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:53.406231 28097 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.61:45099
I20260923 23:09:53.406348 28164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.61:45099 every 8 connection(s)
I20260923 23:09:53.406699 28097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/data/info.pb
I20260923 23:09:53.407356 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 28097
I20260923 23:09:53.407444 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-1/wal/instance
I20260923 23:09:53.407681 5416 external_mini_cluster.cc:1428] Setting key 84193583420347c1dbfda584e3336405
I20260923 23:09:53.407686 28165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:53.409502 28165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.74.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.74.60@KRBTEST.COM" created.
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:53.423389 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.410671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.5.74.62:45951)
0923 23:09:53.410736 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.410738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.410916 (+ 178us) server_negotiation.cc:407] Connection header received
0923 23:09:53.411545 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.411547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.411567 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.411646 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.412464 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.413041 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.414286 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.414651 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.414751 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.419722 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.419724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.419724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.419735 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.421610 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.422245 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.422246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.422247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.422304 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.422647 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.422648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.422648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.422765 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.422822 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.423238 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.423313 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 23:09:53.423709 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.410451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:45951 (local address 127.0.0.1:40824)
0923 23:09:53.410741 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.410800 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:53.411410 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:53.411743 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:53.411747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:53.412038 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:53.412353 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.412359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.413187 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.413188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.414164 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.414167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.414778 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.414779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.414986 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.415594 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:53.415604 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:53.419607 (+ 4003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:53.421726 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.421735 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.421738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.422149 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.422382 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.422383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.422383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.422569 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.422853 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:53.422860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:53.423055 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 23:09:53.423198 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 23:09:53.423411 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":141,"threads_started":1}
I20260923 23:09:53.424441 28165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
I20260923 23:09:53.435127 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.424731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:45099 (local address 127.0.0.1:60164)
0923 23:09:53.424793 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.424808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:53.424861 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:53.425093 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:53.425095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:53.425286 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:53.425361 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.425363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.426367 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.426368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.427353 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.427356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.427885 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.427886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.427976 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.428538 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:53.428543 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:53.431787 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:53.433947 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.433948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.433948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.434184 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.434397 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.434398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.434398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.434453 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.434851 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:53.434852 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:53.434891 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 23:09:53.434949 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:09:53.435026 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 23:09:53.435323 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.424769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.5.74.61:45099)
0923 23:09:53.424929 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.424934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.424954 (+ 20us) server_negotiation.cc:407] Connection header received
0923 23:09:53.424977 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.424979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.424993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.425064 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.425481 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.426269 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.427452 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.427795 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.427862 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.431889 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.431898 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.431898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.431906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.433865 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.434268 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.434271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.434272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.434332 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.434525 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.434526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.434526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.434778 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.434853 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.435014 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.435125 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":138,"threads_started":1}
I20260923 23:09:53.435999 28165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:53.436419 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.436299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60510)
0923 23:09:53.436349 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.436399 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 23:09:53.436601 28116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
I20260923 23:09:53.437065 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.436981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60520)
0923 23:09:53.437012 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.437036 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260923 23:09:53.437173 28165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:40503: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.74.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab.
Entry for principal kudu/127.5.74.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab.
I20260923 23:09:53.439052 5416 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcA05g1/build/release/bin/kudu
/tmp/dist-test-taskcA05g1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.60:40503
--webserver_interface=127.5.74.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab
--principal=kudu/127.5.74.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.74.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.74.62:45951,127.5.74.61:45099,127.5.74.60:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:09:53.468477 28090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } attempt: 3
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
I20260923 23:09:53.476315 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.469002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.5.74.61:45099)
0923 23:09:53.469085 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.469088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.469100 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:09:53.469160 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.469161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.469176 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.469252 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.469601 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.470143 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.470948 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.471191 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.471275 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.473458 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.473460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.473460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.473476 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.475046 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.475458 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.475459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.475460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.475519 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.475721 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.475722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.475722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.475846 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.475924 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.476097 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.476212 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 23:09:53.476315 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.468896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:45099 (local address 127.0.0.1:60180)
0923 23:09:53.468998 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.469010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:53.469087 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:53.469259 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:53.469260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:53.469438 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:53.469504 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.469505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.470234 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.470235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.470820 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.470822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.471285 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.471286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.471379 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.471919 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:53.471924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:53.473340 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:53.475170 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.475171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.475172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.475358 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.475589 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.475590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.475590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.475654 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.475916 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:53.475917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:53.475975 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 23:09:53.476060 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 23:09:53.476209 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 23:09:53.476724 28090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:53.477133 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.476987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60532)
0923 23:09:53.477059 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.477099 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 23:09:53.477571 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.477461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60538)
0923 23:09:53.477511 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.477540 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 23:09:53.477717 28090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:40503: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
I20260923 23:09:53.486482 28165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } attempt: 1
I20260923 23:09:53.487144 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.486952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60548)
0923 23:09:53.487045 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.487098 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 23:09:53.487596 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.487497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60550)
0923 23:09:53.487547 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.487575 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:09:53.487742 28165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:40503: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
I20260923 23:09:53.508883 28090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } attempt: 1
I20260923 23:09:53.509436 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.509279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60560)
0923 23:09:53.509357 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.509401 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 23:09:53.509930 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.509807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60570)
0923 23:09:53.509873 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.509905 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 23:09:53.510042 28090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:40503: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
W20260923 23:09:53.524376 28172 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:09:53.524602 28172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:09:53.524621 28172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:09:53.524642 28172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:09:53.525768 28172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:09:53.525802 28172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:09:53.525816 28172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:09:53.525828 28172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:09:53.525840 28172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:09:53.527247 28172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.74.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.74.62:45951,127.5.74.61:45099,127.5.74.60:40503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/krb5kdc/kudu_127.5.74.60.keytab
--principal=kudu/127.5.74.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.74.60:40503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.74.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:09:53.527446 28172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:09:53.527629 28172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:09:53.532733 28180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:53.532744 28172 server_base.cc:1034] running on GCE node
W20260923 23:09:53.532711 28178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:09:53.532714 28177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:09:53.533282 28172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:09:53.533542 28172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:09:53.534740 28172 hybrid_clock.cc:648] HybridClock initialized: now 1790204993534722 us; error 35 us; skew 500 ppm
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:09:53.537499 28172 init.cc:377] Logged in from keytab as kudu/127.5.74.60@KRBTEST.COM (short username kudu)
I20260923 23:09:53.537974 28172 webserver.cc:460] Webserver started at http://127.5.74.60:34419/ using document root <none> and password file <none>
I20260923 23:09:53.538208 28172 fs_manager.cc:359] Metadata directory not provided
I20260923 23:09:53.538277 28172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:09:53.538450 28172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:09:53.539535 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/data/instance:
uuid: "34f24ffec9fc4a1786c40e39ff04d81d"
format_stamp: "Formatted at 2026-09-23 23:09:53 on dist-test-slave-j4c5"
server_key: "e561319e0f02e0f8a458b85fbfb100f1"
server_key_iv: "13090a3c454cc763a69c0ef8f0fbb925"
server_key_version: "encryptionkey@0"
I20260923 23:09:53.539922 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/wal/instance:
uuid: "34f24ffec9fc4a1786c40e39ff04d81d"
format_stamp: "Formatted at 2026-09-23 23:09:53 on dist-test-slave-j4c5"
server_key: "e561319e0f02e0f8a458b85fbfb100f1"
server_key_iv: "13090a3c454cc763a69c0ef8f0fbb925"
server_key_version: "encryptionkey@0"
I20260923 23:09:53.541358 28172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:09:53.542393 28187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:09:53.542616 28172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:09:53.542721 28172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/data,/tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/wal
uuid: "34f24ffec9fc4a1786c40e39ff04d81d"
format_stamp: "Formatted at 2026-09-23 23:09:53 on dist-test-slave-j4c5"
server_key: "e561319e0f02e0f8a458b85fbfb100f1"
server_key_iv: "13090a3c454cc763a69c0ef8f0fbb925"
server_key_version: "encryptionkey@0"
I20260923 23:09:53.542811 28172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:09:53.555922 28165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } attempt: 2
I20260923 23:09:53.556432 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.556301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60582)
0923 23:09:53.556360 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.556402 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 23:09:53.556768 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.556676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60590)
0923 23:09:53.556720 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.556745 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 23:09:53.556854 28165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.74.60:40503: Network error: Client connection negotiation failed: client connection to 127.5.74.60:40503: connect: Connection refused (error 111)
I20260923 23:09:53.561873 28172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:09:53.564404 28172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:09:53.564574 28172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:09:53.568727 28172 rpc_server.cc:307] RPC server started. Bound to: 127.5.74.60:40503
I20260923 23:09:53.568900 28239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.74.60:40503 every 8 connection(s)
I20260923 23:09:53.569165 28172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/data/info.pb
I20260923 23:09:53.570190 28240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:09:53.571810 28240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:53.574365 5416 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcA05g1/build/release/bin/kudu as pid 28172
I20260923 23:09:53.574471 5416 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcA05g1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204714633713-5416-0/minicluster-data/master-2/wal/instance
I20260923 23:09:53.574703 5416 external_mini_cluster.cc:1428] Setting key cf4b1bb42528cad28e729275959b2adb
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
I20260923 23:09:53.582621 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.572625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.5.74.62:45951)
0923 23:09:53.572747 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.572750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.572944 (+ 194us) server_negotiation.cc:407] Connection header received
0923 23:09:53.573469 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.573470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.573482 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.573547 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.574188 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.575131 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.576468 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.576797 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.576880 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.579258 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.579260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.579260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.579268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.580926 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.581565 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.581566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.581571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.581639 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.581910 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.581911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.581912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.582058 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.582135 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.582430 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.582521 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 23:09:53.582839 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.572593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.62:45951 (local address 127.0.0.1:40838)
0923 23:09:53.572801 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.572850 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:53.573356 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:53.573623 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:53.573627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:53.573899 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:53.574089 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.574093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.575266 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.575267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.576378 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.576382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.576919 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.576920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.577023 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.577587 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:53.577597 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:53.579144 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:53.581035 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.581042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.581045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.581410 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.581713 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.581713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.581714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.581836 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.582136 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:53.582144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:53.582290 (+ 146us) client_negotiation.cc:769] Sending connection context
0923 23:09:53.582397 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 23:09:53.582585 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260923 23:09:53.583381 28240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:53.584141 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.575128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.5.74.60:40503)
0923 23:09:53.575315 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.575320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.575348 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:09:53.575435 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.575439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.575458 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.575556 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.576312 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.577328 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.578173 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.578458 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.578515 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.581084 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.581091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.581092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.581099 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.582660 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.583116 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.583119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.583119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.583184 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.583385 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.583386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.583387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.583621 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.583756 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.583897 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.583998 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
I20260923 23:09:53.585175 28090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } attempt: 2
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.62@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.62@KRBTEST.COM
I20260923 23:09:53.591488 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.583664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.61:45099 (local address 127.0.0.1:60182)
0923 23:09:53.583824 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.583835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:53.583894 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:53.584077 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:53.584078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:53.584350 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:53.584406 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.584409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.585195 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.585196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.585933 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.585935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.586448 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.586449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.586521 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.586968 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:53.586973 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:53.588405 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:53.590313 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.590314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.590315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.590526 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.590726 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.590727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.590728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.590794 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.591073 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:53.591074 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:53.591121 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 23:09:53.591264 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 23:09:53.591397 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260923 23:09:53.591488 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.583749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.5.74.61:45099)
0923 23:09:53.583796 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.583798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.583872 (+ 74us) server_negotiation.cc:407] Connection header received
0923 23:09:53.583963 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.583964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.583977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.584008 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.584480 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.585120 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.586015 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.586329 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.586395 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.588485 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.588486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.588487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.588494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.590218 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.590587 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.590588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.590589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.590647 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.590855 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.590856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.590857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.591010 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.591083 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.591298 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.591385 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:09:53.592541 28240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:09:53.592847 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.585438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.5.74.60:40503)
0923 23:09:53.585470 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.585472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.585483 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:09:53.585560 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.585561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.585570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.585612 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.585889 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.586424 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.587276 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.587527 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.587590 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.589612 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.589613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.589614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.589620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.591425 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.591845 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.591846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.591847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.591900 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.592138 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.592139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.592139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.592316 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.592386 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.592649 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.592732 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:09:53.592913 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.585406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60598)
0923 23:09:53.585450 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.585460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:53.585497 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:53.585626 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:53.585627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:53.585772 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:53.585815 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.585817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.586514 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.586515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.587176 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.587178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.587616 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.587617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.587676 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.588034 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:53.588038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:53.589501 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:53.591525 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.591526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.591527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.591757 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.591991 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.591992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.591992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.592062 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.592476 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:53.592478 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:53.592531 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 23:09:53.592614 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 23:09:53.592789 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 23:09:53.594771 28090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b: Bootstrap starting.
I20260923 23:09:53.595738 28090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:53.596109 28090 log.cc:824] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:53.596921 28090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b: No bootstrap required, opened a new log
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.60@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
I20260923 23:09:53.599197 28090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } }
I20260923 23:09:53.599354 28090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585c18053c34f07ab17656ba70f284b, State: Initialized, Role: FOLLOWER
I20260923 23:09:53.599439 28090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } }
I20260923 23:09:53.599823 28090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:53.599802 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.599961 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:53.601081 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.592821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60610)
0923 23:09:53.593464 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.593474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:53.593556 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:53.593683 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:53.593683 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:53.593816 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:53.593849 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.593850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.594687 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.594688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.595393 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.595395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.595855 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.595856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.595908 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.596569 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:53.596574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:53.598008 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:53.599732 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.599734 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.599735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.600012 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.600263 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.600264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.600264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.600332 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.600682 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:53.600683 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:53.600727 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 23:09:53.600826 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 23:09:53.600946 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620}
I20260923 23:09:53.601143 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.592891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.5.74.60:40503)
0923 23:09:53.592958 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.592960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.593543 (+ 583us) server_negotiation.cc:407] Connection header received
0923 23:09:53.593613 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.593613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.593624 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.593680 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.593913 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.594602 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.595475 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.595772 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.595849 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.598098 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.598100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.598100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.598107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.599642 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.600096 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.600097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.600098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.600205 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.600428 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.600429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.600429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.600605 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.600692 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.600849 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.600999 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":51,"thread_start_us":115,"threads_started":1}
I20260923 23:09:53.602017 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.584945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.5.74.62:45951)
0923 23:09:53.584983 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.584985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.585019 (+ 34us) server_negotiation.cc:407] Connection header received
0923 23:09:53.585090 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.585091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.585099 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.585170 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.585826 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.586547 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.587496 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.587747 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.587826 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.590804 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.590805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.590806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.590812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.592960 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.593374 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.593375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.593376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.596422 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.601373 (+ 4951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.601375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.601375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.601546 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.601764 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.601808 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.601901 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 23:09:53.602485 28258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:53.602545 28258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:53.602694 28240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d: Bootstrap starting.
I20260923 23:09:53.603412 28240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:53.603698 28240 log.cc:824] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:53.604849 28240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d: No bootstrap required, opened a new log
I20260923 23:09:53.606300 28240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } }
I20260923 23:09:53.606426 28240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f24ffec9fc4a1786c40e39ff04d81d, State: Initialized, Role: FOLLOWER
I20260923 23:09:53.606510 28240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } }
I20260923 23:09:53.606837 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.606904 28240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:53.606922 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:09:53.608731 28272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:53.609340 28272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:53.609313 28022 master_runner.cc:186] Error getting master registration for 127.5.74.61:45099: OK, instance_id { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" instance_seqno: 1790204993404787 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:09:53.611794 28022 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:09:53.611160 28172 master_runner.cc:186] Error getting master registration for 127.5.74.61:45099: OK, instance_id { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" instance_seqno: 1790204993404787 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:09:53.614472 28172 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.74.61@KRBTEST.COM
I20260923 23:09:53.621022 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.608469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.5.74.61:45099)
0923 23:09:53.608691 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.608694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.608708 (+ 14us) server_negotiation.cc:407] Connection header received
0923 23:09:53.608733 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.608734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.608746 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.608827 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.612306 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.613064 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.614141 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.614458 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.614502 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.617824 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.617826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.617826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.617834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.619605 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.620005 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.620006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.620007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.620065 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.620317 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.620318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.620318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.620455 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.620605 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.620826 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.620916 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260923 23:09:53.623318 5416 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:09:53.624320 28064 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7585c18053c34f07ab17656ba70f284b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40848
I20260923 23:09:53.624502 28064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 23:09:53.624560 28064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:53.625562 28064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } }
I20260923 23:09:53.625818 28064 leader_election.cc:288] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [CANDIDATE]: Term 1 election: Requested vote from peers fbd3d9bd2bbf41a28853416625020c5a (127.5.74.61:45099), 34f24ffec9fc4a1786c40e39ff04d81d (127.5.74.60:40503)
I20260923 23:09:53.626166 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7585c18053c34f07ab17656ba70f284b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "34f24ffec9fc4a1786c40e39ff04d81d"
I20260923 23:09:53.626338 28214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:53.627322 28214 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7585c18053c34f07ab17656ba70f284b in term 1.
I20260923 23:09:53.627560 28041 leader_election.cc:302] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34f24ffec9fc4a1786c40e39ff04d81d, 7585c18053c34f07ab17656ba70f284b; no voters:
I20260923 23:09:53.627761 28247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:09:53.627918 28247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [term 1 LEADER]: Becoming Leader. State: Replica: 7585c18053c34f07ab17656ba70f284b, State: Running, Role: LEADER
I20260923 23:09:53.628032 28247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } }
I20260923 23:09:53.628688 28281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7585c18053c34f07ab17656ba70f284b. Latest consensus state: current_term: 1 leader_uuid: "7585c18053c34f07ab17656ba70f284b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.628906 28281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:53.629243 28283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:09:53.629624 28283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:09:53.631826 28214 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [term 1 FOLLOWER]: Refusing update from remote peer 7585c18053c34f07ab17656ba70f284b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:53.632279 28281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:53.634171 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7585c18053c34f07ab17656ba70f284b. Latest consensus state: current_term: 1 leader_uuid: "7585c18053c34f07ab17656ba70f284b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.634292 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:53.635223 28281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7585c18053c34f07ab17656ba70f284b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.635325 28281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:53.635483 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7585c18053c34f07ab17656ba70f284b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.635586 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:53.635711 28284 mvcc.cc:205] Tried to move back new op lower bound from 7332679653913006080 to 7332679653901225984. Current Snapshot: MvccSnapshot[applied={T|T < 7332679653913006080}]
I20260923 23:09:53.635834 28285 mvcc.cc:205] Tried to move back new op lower bound from 7332679653913006080 to 7332679653901225984. Current Snapshot: MvccSnapshot[applied={T|T < 7332679653913006080}]
I20260923 23:09:53.636648 28283 catalog_manager.cc:1347] Generated new cluster ID: 8de98d3b85644177889442bd13ac7ba2
I20260923 23:09:53.636708 28283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:09:53.648326 28165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } attempt: 3
I20260923 23:09:53.649360 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.627106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.5.74.61:45099)
0923 23:09:53.627202 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.627204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.632302 (+ 5098us) server_negotiation.cc:407] Connection header received
0923 23:09:53.632412 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.632413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.632426 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.632461 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.640797 (+ 8336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.641577 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.643280 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.643613 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.643684 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.645954 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.645968 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.645968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.645976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.647940 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.648405 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.648406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.648407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.648472 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.648712 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.648713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.648714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.648864 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.648959 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.649173 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.649255 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 23:09:53.651687 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.627121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.5.74.60:40503)
0923 23:09:53.627163 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.627166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.640895 (+ 13729us) server_negotiation.cc:407] Connection header received
0923 23:09:53.640995 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.640996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.641008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.641037 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.641761 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.642494 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.644557 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.644880 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.644942 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.647745 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.647748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.647748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.647758 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.649618 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.650520 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.650521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.650522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.650585 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.650923 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.650924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.650924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.651052 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.651135 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.651492 (+ 357us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.651587 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 23 23:09:53 dist-test-slave-j4c5 krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790204993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.74.61@KRBTEST.COM for kudu/127.5.74.60@KRBTEST.COM
I20260923 23:09:53.660399 28283 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:09:53.660887 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.648609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.74.60:40503 (local address 127.0.0.1:60626)
0923 23:09:53.648960 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 23:09:53.648975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 23:09:53.649021 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:09:53.649999 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:09:53.650000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:09:53.650173 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:09:53.650218 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.650220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.651415 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.651416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.652733 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:09:53.652736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.653336 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.653337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:09:53.653399 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.653947 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:09:53.653952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:09:53.655565 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:09:53.658561 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.658562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.658563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.658793 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.659824 (+ 1031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:09:53.659825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:09:53.659825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:09:53.659907 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:09:53.660549 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:09:53.660551 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:09:53.660614 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 23:09:53.660678 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:09:53.660782 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328}
I20260923 23:09:53.661770 28245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.649397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.5.74.60:40503)
0923 23:09:53.649451 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.649453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.649461 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:09:53.649482 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.649482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.649510 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.649582 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.650427 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.651297 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.652848 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.653217 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.653282 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.655675 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.655676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.655677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.655685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.658388 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.659659 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.659660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.659661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.659742 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.660000 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.660001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.660001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.660333 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.661295 (+ 962us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.661413 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.661559 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:09:53.662619 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:53.627723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.5.74.62:45951)
0923 23:09:53.627770 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.627772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.644664 (+ 16892us) server_negotiation.cc:407] Connection header received
0923 23:09:53.644781 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.644782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.644794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.644824 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.649661 (+ 4837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.650429 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.651426 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:53.651698 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:53.651745 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:53.657646 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:09:53.657648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:09:53.657649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:09:53.657657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:09:53.659579 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.661389 (+ 1810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.661391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.661391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.661491 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:09:53.661719 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:09:53.661719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:09:53.661720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:09:53.661861 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:09:53.661947 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:53.662413 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 23:09:53.662519 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:09:53.661226 28283 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:09:53.663625 28165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a: Bootstrap starting.
I20260923 23:09:53.664397 28165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a: Neither blocks nor log segments found. Creating new log.
I20260923 23:09:53.664697 28165 log.cc:824] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a: Log is configured to *not* fsync() on all Append() calls
I20260923 23:09:53.665925 28165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a: No bootstrap required, opened a new log
I20260923 23:09:53.667924 28165 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } }
I20260923 23:09:53.668058 28165 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd3d9bd2bbf41a28853416625020c5a, State: Initialized, Role: FOLLOWER
I20260923 23:09:53.668133 28165 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } }
I20260923 23:09:53.668716 28165 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:09:53.669962 28139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7585c18053c34f07ab17656ba70f284b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fbd3d9bd2bbf41a28853416625020c5a"
I20260923 23:09:53.670061 28139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:09:53.670909 28139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7585c18053c34f07ab17656ba70f284b in term 1.
I20260923 23:09:53.671264 28138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [term 1 FOLLOWER]: Refusing update from remote peer 7585c18053c34f07ab17656ba70f284b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:09:53.671576 28281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:09:53.673722 28302 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7585c18053c34f07ab17656ba70f284b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.673883 28302 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:53.673983 28302 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7585c18053c34f07ab17656ba70f284b. Latest consensus state: current_term: 1 leader_uuid: "7585c18053c34f07ab17656ba70f284b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.674082 28302 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:53.674150 28302 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7585c18053c34f07ab17656ba70f284b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.674242 28302 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:09:53.674983 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7585c18053c34f07ab17656ba70f284b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7585c18053c34f07ab17656ba70f284b" member_type: VOTER last_known_addr { host: "127.5.74.62" port: 45951 } } peers { permanent_uuid: "fbd3d9bd2bbf41a28853416625020c5a" member_type: VOTER last_known_addr { host: "127.5.74.61" port: 45099 } } peers { permanent_uuid: "34f24ffec9fc4a1786c40e39ff04d81d" member_type: VOTER last_known_addr { host: "127.5.74.60" port: 40503 } } }
I20260923 23:09:53.675081 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b [sys.catalog]: This master's current role is: LEADER
I20260923 23:09:53.676779 28303 mvcc.cc:205] Tried to move back new op lower bound from 7332679653980430336 to 7332679653901225984. Current Snapshot: MvccSnapshot[applied={T|T < 7332679653913006080}]
W20260923 23:09:53.677335 28301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:09:53.677448 28301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:09:53.677515 28283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7585c18053c34f07ab17656ba70f284b: Generated new TSK 0
I20260923 23:09:53.677716 28283 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:09:53.704792 28245 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.703230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.5.74.60:40503)
0923 23:09:53.703280 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.703282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.703982 (+ 700us) server_negotiation.cc:407] Connection header received
0923 23:09:53.704143 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.704144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.704176 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.704214 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.704716 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60630: BlockingRecv error: recv got EOF from 127.0.0.1:60630 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260923 23:09:53.705569 28093 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.703618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.5.74.62:45951)
0923 23:09:53.703674 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.703676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.705051 (+ 1375us) server_negotiation.cc:407] Connection header received
0923 23:09:53.705151 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.705153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.705164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.705194 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.705506 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40860: BlockingRecv error: recv got EOF from 127.0.0.1:40860 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260923 23:09:53.706722 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.705983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.5.74.61:45099)
0923 23:09:53.706022 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.706023 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.706136 (+ 113us) server_negotiation.cc:407] Connection header received
0923 23:09:53.706229 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.706230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.706239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.706268 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.706649 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60192: BlockingRecv error: recv got EOF from 127.0.0.1:60192 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:09:53.709229 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.708485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.5.74.61:45099)
0923 23:09:53.708521 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.708523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.708688 (+ 165us) server_negotiation.cc:407] Connection header received
0923 23:09:53.708793 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.708793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.708811 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.708840 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.709182 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60202: BlockingRecv error: recv got EOF from 127.0.0.1:60202 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:09:53.712280 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.711490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.5.74.61:45099)
0923 23:09:53.711522 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.711524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.711722 (+ 198us) server_negotiation.cc:407] Connection header received
0923 23:09:53.711815 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.711815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.711825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.711854 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.712219 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60212: BlockingRecv error: recv got EOF from 127.0.0.1:60212 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 23:09:53.716503 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.715679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.5.74.61:45099)
0923 23:09:53.715716 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.715718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.715938 (+ 220us) server_negotiation.cc:407] Connection header received
0923 23:09:53.716035 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.716036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.716046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.716099 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.716455 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60220: BlockingRecv error: recv got EOF from 127.0.0.1:60220 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 23:09:53.722656 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.721808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.5.74.61:45099)
0923 23:09:53.721848 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.721850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.722080 (+ 230us) server_negotiation.cc:407] Connection header received
0923 23:09:53.722183 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.722184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.722195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.722224 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.722599 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60230: BlockingRecv error: recv got EOF from 127.0.0.1:60230 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 23:09:53.733659 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.732500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.5.74.61:45099)
0923 23:09:53.732558 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.732561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.732954 (+ 393us) server_negotiation.cc:407] Connection header received
0923 23:09:53.733099 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.733101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.733114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.733179 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.733591 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60232: BlockingRecv error: recv got EOF from 127.0.0.1:60232 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260923 23:09:53.752163 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.751125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.5.74.61:45099)
0923 23:09:53.751173 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.751176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.751421 (+ 245us) server_negotiation.cc:407] Connection header received
0923 23:09:53.751527 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.751528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.751540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.751570 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.751988 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60240: BlockingRecv error: recv got EOF from 127.0.0.1:60240 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 23:09:53.786916 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.785919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.5.74.61:45099)
0923 23:09:53.785970 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.785972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.786255 (+ 283us) server_negotiation.cc:407] Connection header received
0923 23:09:53.786372 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.786373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.786385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.786418 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.786848 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60242: BlockingRecv error: recv got EOF from 127.0.0.1:60242 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 23:09:53.853610 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.852626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.5.74.61:45099)
0923 23:09:53.852756 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.852759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.852883 (+ 124us) server_negotiation.cc:407] Connection header received
0923 23:09:53.853024 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.853025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.853039 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.853127 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.853532 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60250: BlockingRecv error: recv got EOF from 127.0.0.1:60250 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260923 23:09:53.983968 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:53.983072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.5.74.61:45099)
0923 23:09:53.983191 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:53.983193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:53.983294 (+ 101us) server_negotiation.cc:407] Connection header received
0923 23:09:53.983400 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:53.983401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:53.983412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:53.983496 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:53.983880 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60256: BlockingRecv error: recv got EOF from 127.0.0.1:60256 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260923 23:09:54.242105 28170 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:09:54.241398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.5.74.61:45099)
0923 23:09:54.241474 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:54.241476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:54.241486 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:09:54.241551 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:54.241552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:54.241564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:54.241643 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:09:54.242001 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60264: BlockingRecv error: recv got EOF from 127.0.0.1:60264 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 23:09:54.609840 28272 catalog_manager.cc:1259] Loaded cluster ID: 8de98d3b85644177889442bd13ac7ba2
I20260923 23:09:54.609925 28272 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d: loading cluster ID for follower catalog manager: success
I20260923 23:09:54.610839 28272 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d: acquiring CA information for follower catalog manager: success
I20260923 23:09:54.611182 28272 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34f24ffec9fc4a1786c40e39ff04d81d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:09:54.677925 28301 catalog_manager.cc:1259] Loaded cluster ID: 8de98d3b85644177889442bd13ac7ba2
I20260923 23:09:54.678005 28301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a: loading cluster ID for follower catalog manager: success
I20260923 23:09:54.678896 28301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a: acquiring CA information for follower catalog manager: success
I20260923 23:09:54.679248 28301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbd3d9bd2bbf41a28853416625020c5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:09:54.758167 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:54.755394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.5.74.61:45099)
0923 23:09:54.755584 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:54.755586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:54.755595 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:09:54.755622 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:54.755623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:54.755672 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:54.755751 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:54.755926 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.756518 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.757284 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.757519 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.757577 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:54.757743 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:54.757845 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:54.757904 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:54.758019 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 23:09:54.758038 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260923 23:09:54.762681 28381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:54.760067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.5.74.60:40503)
0923 23:09:54.760199 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:54.760201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:54.760218 (+ 17us) server_negotiation.cc:407] Connection header received
0923 23:09:54.760293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:54.760294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:54.760331 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:54.760407 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:54.760606 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.761089 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.761948 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.762149 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.762205 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:54.762335 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:54.762474 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:54.762529 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:54.762596 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 23:09:54.762614 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260923 23:09:54.768132 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:54.764683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.5.74.62:45951)
0923 23:09:54.765054 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:54.765056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:54.765063 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:09:54.765108 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:54.765108 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:54.765164 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:54.765246 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:54.765449 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.766234 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.767091 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.767297 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.767401 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:54.767679 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:54.767799 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:54.767860 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:54.768018 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 23:09:54.768046 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":45,"threads_started":1}
I20260923 23:09:54.768208 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:54.764727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.5.74.61:45099)
0923 23:09:54.764779 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:54.764781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:54.764964 (+ 183us) server_negotiation.cc:407] Connection header received
0923 23:09:54.765028 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:54.765028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:54.765037 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:54.765079 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:54.765284 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.766144 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.767317 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.767639 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.767681 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:54.767893 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:54.767940 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:54.767990 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:54.768077 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 23:09:54.768102 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:09:54.768734 28381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:09:54.764776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.5.74.60:40503)
0923 23:09:54.764824 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:09:54.764826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:09:54.765401 (+ 575us) server_negotiation.cc:407] Connection header received
0923 23:09:54.765464 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:09:54.765465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:09:54.765475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:09:54.765627 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:09:54.765906 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.766502 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.767484 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:09:54.767727 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:09:54.767801 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:09:54.767940 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:09:54.767973 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:09:54.768029 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 23:09:54.768610 (+ 581us) server_negotiation.cc:299] Negotiation successful
0923 23:09:54.768666 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:09:54.778235 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 28022
I20260923 23:09:54.783865 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 28097
I20260923 23:09:54.788888 5416 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcA05g1/build/release/bin/kudu with pid 28172
2026-09-23T23:09:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2918 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5824 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280158 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST